There’s a very spe­cific rep­u­ta­tion I want to have on a team: “Nat helps me solve my prob­lems. Nat get things I care about done.”

There’s a very spe­cific rep­u­ta­tion I want to have on a team: “Nat helps me solve my prob­lems. Nat get things I care about done.”

I keep a bul­let jour­nal. I’m not one of those peo­ple you see on Pinterest with the fancy spreads — I mostly just use black ink, the stan­dard setup, and the oc­ca­sional cus­tom col­lec­tion.

Every time I join a new team, I go to the next fresh page, and on top of that page I write: “WTF - [Team Name].” Then I make a note every time I run into some­thing that makes me go “wtf,” and a task every time I come up with some­thing I want to change.

For two weeks, that’s all I do. I just write it down. I don’t tell the team every­thing that I think they’re do­ing wrong. I don’t show up at retro with all the stuff I think they need to change. I just watch, and lis­ten, and I write down every­thing that seems deeply weird.

This is a trick I picked up from a team lead a few years ago, who learned it from a pre­vi­ous lead of his in turn. It’s one of my most pow­er­ful tech­niques for mak­ing changes on a team, and man­ag­ing my­self while I do it. So I’m go­ing to walk you through how I use that list, and how it helps me to build a rep­u­ta­tion as some­one who’s re­ally ef­fec­tive at get­ting stuff done, and avoid be­ing some­one who’s com­plain­ing all the time.

There’s al­ways stuff that makes me go “wtf” on a new team. The team talks for an hour in retro about a se­ri­ous prob­lem, and then leaves with­out mak­ing any ac­tion items. The tests don’t run lo­cally and no one seems to no­tice. Big chunks of the build board are al­ways red. Only one per­son can do some crit­i­cal, time-sen­si­tive thing. The team is spend­ing a bunch of time on some fea­ture, but when I ask around no one can seems to know why it’s im­por­tant or how it’ll help a cus­tomer.

Once I’ve got a nice big list, I start cross­ing things off. There are four rea­sons at this point that I might cross off some­thing I’ve put on that list:

There’s ac­tu­ally a good rea­son for it­The team is al­ready work­ing on a fix­The team does­n’t care about itIt’s re­ally easy to fix

If the tests don’t run lo­cally, for in­stance, that might be a known is­sue that there’s an on­go­ing ef­fort to ad­dress. The team might do all of their work on vir­tual ma­chines, and have a sim­ple chat com­mand that pro­vi­sions those ma­chines for them. Or they might have a pretty good con­tin­u­ous in­te­gra­tion sys­tem and good habits around mak­ing small changes, so not be­ing able to run the tests lo­cally is­n’t stop­ping them from de­ploy­ing mul­ti­ple times a day.

Sometimes, it’ll turn out that there’s a re­ally sim­ple fix for some of the things I’ve iden­ti­fied. Maybe there’s some doc­u­men­ta­tion I can write, once I know where it is, or maybe there’s an easy change once I find the right scripts. That’s not al­ways im­me­di­ately ob­vi­ous when I first see a prob­lem. When I do see an easy fix, though, I’ll just go ahead and make it.

After a few weeks, though, I’ll still have a bunch of weird, un­re­solved is­sues on that list. At this point I’ll start talk­ing about it with other peo­ple on the team, the team lead, and my man­ager.

I’ll ask why things on the list are that way, and how they got to be that way. I’m try­ing to es­tab­lish cred­i­bil­ity as some­one who’s gen­uinely cu­ri­ous and em­pa­thetic, who’s pa­tient, and who re­spects the ex­per­tise of my cowork­ers. That’s the rep­u­ta­tion that’s go­ing to let me make changes later.

Generally, I’ll find out that the things that prob­lems I’ve no­ticed are around for one of a few rea­sons.

The team has got­ten used to it­The prob­lem is rel­a­tively new, and the old prob­lem it re­placed was much worseThey don’t know how to fix the prob­lemThey’ve tried to fix the prob­lem be­fore and failed

On a lot of teams, when I ask some ques­tions about things that turn out to be in the first few ques­tions, the per­son I ask will just fix them im­me­di­ately. Or they’ll help me fig­ure out how to fix them. If it’s a tech­ni­cal prob­lem, that means writ­ing a story or a ticket to­gether, and then we’ll work on it. If it’s more process or so­cial, it means bring­ing the prob­lem up at retro and talk­ing about it with the whole team.

At this point I’m look­ing for one or two prob­lems that have been bug­ging one of my new team­mates for a while, and that have rel­a­tively sim­ple so­lu­tions. I’m look­ing for some­thing I can put on the retro board and know I won’t be the only per­son who’s both­ered by that prob­lem. Then, dur­ing the team con­ver­sa­tion about the prob­lem, I’ll iden­tify some­thing that team­mate sug­gests as an ac­tion item that we could try im­me­di­ately. That way the team starts to see me as some­one who helps them solve their prob­lems.

The feel­ing that I want to cre­ate, the as­so­ci­a­tion I want peo­ple to have with me, is, “Oh, Nat joined the team and lit­tle things started to get bet­ter, al­most im­me­di­ately. It feels like we’re start­ing to make some progress. And it’s not like they showed up and started telling me what to do, ei­ther. They’re re­ally lis­ten­ing to me, they’re help­ing me ex­plain my­self to the rest of the team.”

Pretty soon, I’ll start to get in to the re­ally sticky is­sues. The prob­lems the team knows about but is afraid of deal­ing with. The things that aren’t “that bad,” but that no one wants to talk about. Maybe they’re miss­ing the tech­ni­cal skills to deal with the prob­lem. Maybe there’s a knotty peo­ple prob­lem at the cen­ter of it.

At this point I’m go­ing to be talk­ing to my man­ager. I’m go­ing to bring them that list I’ve been work­ing on, and I’m go­ing to say some­thing like, “Now that I’ve been on the team for a few weeks, this is what I’m see­ing. We’re mak­ing progress on some of it, but some of these seem like they’re go­ing to take longer. I wanted to get your thoughts be­fore I try to do any­thing about them. Is there some­thing I’m miss­ing? Is there a par­tic­u­lar area I’d like you to fo­cus?”

The re­ac­tion I’m look­ing for from my man­ager, at this point, is some­thing like, “Wow. This is re­ally val­i­dat­ing. I’ve been con­cerned about these things but the team does­n’t seem re­ally both­ered by them, so I did­n’t want to push too hard. I’m glad you’re bring­ing this up.”

Then we can have a con­ver­sa­tion about what their con­cerns and prob­lems are. I can do some re­flec­tive lis­ten­ing to help them or­ga­nize their thoughts, and I can talk about what I’ve seen work well, or not, in the past. They’ll start to see me as some­one with good judge­ment, and some­one they can come to for help solv­ing their harder prob­lems.

There’s a very spe­cific rep­u­ta­tion I want to have on a team: “Nat helps solve my prob­lems. Nat get things I care about done.” That’s the rep­u­ta­tion that’s go­ing to get me the re­sults I want in next year’s per­for­mance re­view. That’s the rep­u­ta­tion that’s go­ing to get me a re­fer­ral a few years from now.

Before I started keep­ing this kind of list, I brought up prob­lem I saw im­me­di­ately, as soon as I no­ticed it. The rep­u­ta­tion I got was, “Nat’s al­ways com­plain­ing about things. Nat thinks we’re never do­ing things right.” People stopped lis­ten­ing to me. I was per­son­ally frus­trated, and pro­fes­sion­ally in­ef­fec­tive.

There’s no faster way to to­tally sink my cred­i­bil­ity, as a new team mem­ber, by mak­ing a huge fuss over some­thing that’s not a prob­lem, or that the team does­n’t see as a prob­lem, or that there’s al­ready an ef­fort to fix, or that there’s a re­ally sim­ple way to fix that I just did­n’t see at first. There are al­ways so many prob­lems on a team, so many things that could be bet­ter, that I’m only ever go­ing to solve a hand­ful of them. Working on prob­lems in the or­der I no­ticed them is rarely the most ef­fec­tive or­der. So the WTF Notebook gives me a place to park the im­pulse to fix it now, damn it! un­til I have more con­text for de­cid­ing what to work on first.

Instead, for two weeks, I just write things down.

Periodic re­minder that Code for America is usu­ally hir­ing, and they pair and write tests. Until the end of this month they have a Software Engineer role up for a team that works San Francisco hours. If you’re look­ing for a “show up, write code, go home” ex­pe­ri­ence, and want to help Americans ac­cess food stamps and other safety net ser­vices, this is a team that can de­liver it — es­pe­cially if you have some ex­pe­ri­ence with Rails or Spring.

If, on the other hand, you’re in­ter­ested in gnarly cloud in­fra­struc­ture and soft­ware prob­lems for the Department of Defense, check out Rise8. If you’ve heard about Kessel Run, or Pivotal’s work with the Air Force gen­er­ally, Rise8 is where many of those folks ended up. They also prac­tice de­sign think­ing, test-dri­ven de­vel­op­ment, and con­tin­u­ous de­ploy­ment, but they’re teach­ing them to folks who have never used these prac­tices be­fore, and pair­ing with mil­i­tary ser­vice peo­ple. Their job list­ings men­tion ex­pe­ri­ence at Pivotal Labs by name.

If you’ve got an ac­tive job search run­ning and you’re strug­gling to keep track of it all, check out Davis Frank’s guide to Job Search Journaling with Obsidian.

I’ve men­tioned Seeing Like a State be­fore but I reread it while we were on the road, and, and, man, se­ri­ously, if there’s one book I wish every­one I talk to had read, it’s this one. Nothing ex­plains sys­tems think­ing in ac­tion bet­ter. Nothing has more use­ful anec­dotes for il­lus­trat­ing how large or­ga­ni­za­tions work, and why they work the way they do.

The other book I’ve read by James C. Scott is Against the Grain, and if you’re at all in­ter­ested in the his­tory of the ear­li­est states and the ini­tial de­vel­op­ment of hu­man civ­i­liza­tion, that book will ab­solutely blow your mind.

Ed Zitron’s piece re­cently about How Our Need For Attention Online Drives Us Crazy ar­tic­u­lated a bunch of half-formed thoughts I’ve been chew­ing on and try­ing to fig­ure out how to write about. It does­n’t men­tion Slack ex­plic­itly, but I’ve seen Slack drive a lot of these same processes at work.

Halo 2 in HD: Pushing the Original Xbox to the Limit

Halo 2 in HD: Pushing the Original Xbox to the Limit

This blog post goes over all of the work I’ve done to add HD res­o­lu­tion sup­port to the Original Xbox ver­sion of Halo 2. From patch­ing the game to mod­i­fy­ing the hard­ware of the Xbox con­sole to writ­ing cus­tom tools for per­for­mance bench­mark­ing, my goal with this pro­ject was to push the lim­its of both and see how far I could go. I’ve tried to keep this blog post as short as I could and only in­clude the most tech­ni­cally in­ter­est­ing parts but even then it ended up quite long.

A long time friend who goes by the han­dle “doom” has spent the past few years re­verse en­gi­neer­ing and re­search­ing the hard­ware and soft­ware on the orig­i­nal Xbox. His end goal was to learn more about PC hard­ware and see how far he could push the con­sole. Some of his work in­cludes swap­ping out the stock Pentium 3 CPU run­ning at 733Mhz for a vari­ant of the Pentium 3 CPU run­ning at 1.4Ghz us­ing a cus­tom made CPU in­ter­poser board, and even be­ing able to over­clock it up­wards of ~2Ghz.

Doom also wrote cus­tom patches for the Xbox ker­nel in or­der to on-the-fly patch tim­ing cal­cu­la­tions for games so they ran prop­erly with the faster CPU. Combined with a few other hard­ware up­grades such as ad­di­tional RAM and an SSD, doom started to re­fer to these as “god boxes”. These god boxes were also run­ning a cus­tom ker­nel (or BIOS im­age) that doom made to sup­port all of the hard­ware mod­i­fi­ca­tions and push the hard­ware and soft­ware as far as they could go. One of his demos for his work was show­ing the open­ing se­quence in Half-Life 2 which is no­to­ri­ous for abysmally slow load­ing times and poor per­for­mance on the Xbox, run­ning at a solid 30 FPS and load­ing in a mat­ter of sec­onds. But there were still ad­di­tional ben­e­fits to be had. Doom wanted some­one to cre­ate a proper HD res­o­lu­tion patch for a pop­u­lar game and re­ally uti­lize the hard­ware up­grades he per­formed.

One night while talk­ing over Discord doom asked if I would be in­ter­ested in de­vel­op­ing an HD patch for Halo 2 and in ex­change he would pro­vide me with a god box to de­velop it on. Halo 2 has a max sup­ported video res­o­lu­tion of 480p and patch­ing in sup­port for 720p (and pos­si­bly 1080i) would get a lot of at­ten­tion to demon­strate the ben­e­fits of all this work. We both knew that many of the com­mu­nity “HD” or “720p” game patches were not ac­tu­ally func­tion­ing cor­rectly and that patch­ing in HD res­o­lu­tion sup­port for a game was more work than just search­ing for 640/480 in a dis­as­sem­bler and chang­ing the res­o­lu­tion. These patches re­quire a deep un­der­stand­ing of 3D graph­ics, DirectX APIs, and a lot of spe­cific knowl­edge about the game and Xbox con­sole. Having spent years re­verse en­gi­neer­ing the Xbox and Halo 2’s game en­gine I had the per­fect back­ground to take on the task. As doom would put it “there’s no­body more qual­i­fied than you to do it for halo 2 so that’s why I asked”. While it piqued my in­ter­est (and I was pretty jeal­ous of these god boxes and all the ex­pe­ri­ence he’d got­ten de­vel­op­ing them), I made a re­quest/​re­quire­ment be­fore I would even en­ter­tain the idea.

The up­graded CPU has more than dou­ble the pro­cess­ing power com­pared to the stock CPU, how­ever, the GPU was go­ing to take on most of the in­creased pro­cess­ing load once the video res­o­lu­tion was in­creased. After all, each ad­di­tional pixel in the out­put im­age would re­sult in more pixel shader cal­cu­la­tions which meant more work the GPU would have to do. If he could man­age to over­clock the GPU I would do it, but at stock clock speeds it was­n’t worth the time it would take to de­velop this patch just to have it fall over on the GPU. He said he would look into it, and af­ter a few weeks time he came back and said it was done. He man­aged to over­clock the GPU by ~15%, and said he had the “GENESIS-3” con­sole ready for me (a nick­name for the 3rd it­er­a­tion of the “god box” up­grades he’d been work­ing on).

Having spent the past few years re­verse en­gi­neer­ing and re-im­ple­ment­ing the Halo 2 ren­der­ing en­gine I al­ready had a men­tal list of things I’d need to change to sup­port higher video res­o­lu­tions. The first thing that needed to be changed was the size of the D3D front and back buffers. The setup for the D3D de­vice has 3 func­tions that need to be mod­i­fied in or­der to use the proper res­o­lu­tion for the cur­rent video mode. The first is _rasterizer_detect_video_mode which checks the video mode and sets some global vari­ables for widescreen and pro­gres­sive video modes. Next is _rasterizer_init_screen_bounds which sets up the screen di­men­sions used for cre­at­ing the D3D de­vice, view frus­tum, and a num­ber of other things. Lastly is ras­ter­iz­er_de­vice_ini­tial­ize which is re­spon­si­ble for set­ting up the D3D de­vice. Below is a short­ened ver­sion of these 3 func­tions with the lines of in­ter­est high­lighted. All of the code shown in this post has been re­verse en­gi­neered from as­sem­bly back into C for ease of un­der­stand­ing.

Halo 2 al­ready sup­ports 480p, or does it…

If you’ve ever looked at the back of the game case for Halo 2 you might have seen it sup­ports 480p. However, look­ing at line 42 above, the D3DPRESENTFLAG_PROGRESSIVE flag is not be­ing set on the pre­sent pa­ra­me­ters. And, if we look at the call site for the _rasterizer_init_screen_bounds func­tion we see this:

The scale pa­ra­me­ter is al­ways set to 1.0f, which means the screen_bounds are al­ways set to 640×480 re­gard­less of what the video mode is set to on the con­sole. On the Original Xbox 480p is con­sid­ered to be 720×480, which means that Halo 2 does not ren­der in 480p na­tively re­gard­less of what the video set­tings are set to. If you en­able 480p mode on the con­sole you’ll get a 480p sig­nal out but that’s be­cause af­ter the game is done draw­ing to the 640×480 back buffer it’ll get up-scaled to 720×480 by the GPU be­fore be­ing fed to the video en­coder. I of­ten get com­ments say­ing “that’s not not a 16:9 res­o­lu­tion” or “that’s not real 480p”, but “480p” en­cap­su­lates a range of res­o­lu­tions and as­pect ra­tios and 720×480 is the res­o­lu­tion the Xbox con­sid­ers to be 480p (so take it up with Microsoft, not me…).

If you’ve ever played Halo 2 in 480p mode with wide screen en­abled you may have no­ticed that things look a lit­tle weird. That’s be­cause when wide screen mode is en­abled the game will use an anamor­phic cam­era with an as­pect ra­tio of 1.33:1. That means it ren­ders 1.3x the width into the same 640×480 sur­face as it would when wide screen mode is dis­abled. Here is a com­par­i­son show­ing the ef­fect anamor­phic scal­ing has on the Zanzibar wheel:

I’m not en­tirely sure why it does this and my only guess is if you set your TV to stretch mode it would “cancel out” the hor­i­zon­tal “squish” in­tro­duced by the anamor­phic scal­ing and look some­what nor­mal. However, I per­son­ally hate it and wanted the clean­est im­age I could get out of the con­sole so I added an op­tion to dis­able the anamor­phic scal­ing en­tirely.

To cre­ate the D3D front/​back buffers with the right di­men­sions we’ll need to change g_pro­gres­sive_s­can_en­abled to be set when 720p is en­abled, set the screen_bounds and frame_bounds vari­ables based on the proper video res­o­lu­tion for the video mode set, and fi­nally set some ad­di­tional flags on the D3D pre­sent pa­ra­me­ters de­pend­ing on if the video mode is pro­gres­sive or in­ter­laced (1080i mode). The pseudo code for the mod­i­fi­ca­tions is shown be­low with the changed lines high­lighted. I ig­nored the scale vari­able in _rasterizer_init_screen_bounds be­cause it’s only ever set to 1.0 any­way.

void _rasterizer_detect_video_mode()

DWORD video­Stan­dard = XGetVideoStandard();

DWORD vide­oFlags = XGetVideoFlags();

if (videoStandard == XC_VIDEO_STANDARD_PAL_I)

g_re­fresh_rate_hz = (videoFlags & XC_VIDEO_FLAGS_PAL_60Hz) != 0 ? 60 : 50;

g_let­ter­box_en­abled = (videoFlags & XC_VIDEO_FLAGS_LETTERBOX) != 0;

g_widescreen_en­abled = (videoFlags & XC_VIDEO_FLAGS_WIDESCREEN) != 0;

g_pro­gres­sive_s­can_en­abled = (videoFlags & (XC_VIDEO_FLAGS_HDTV_480p | XC_VIDEO_FLAGS_HDTV_720p)) != 0;

void _rasterizer_init_screen_bounds(int x_off, int y_off, float scale)

// Set de­fault res­o­lu­tion to 640x480.

float width = 640.0f;

float height = 480.0f;

// Adjust res­o­lu­tion based on cur­rent video mode set.

DWORD vide­oFlags = XGetVideoFlags();

if ((videoFlags & XC_VIDEO_FLAGS_HDTV_1080i) != 0)

width = 1920;

height = 1080;

else if ((videoFlags & XC_VIDEO_FLAGS_HDTV_720p) != 0)

width = 1280;

height = 720;

else if ((videoFlags & XC_VIDEO_FLAGS_HDTV_480p) != 0)

width = 720;

ras­ter­iz­er_­glob­als.screen_bounds.x0 = 0;

ras­ter­iz­er_­glob­als.screen_bounds.y0 = 0;

ras­ter­iz­er_­glob­als.screen_bounds.x1 = (int)width;

ras­ter­iz­er_­glob­als.screen_bounds.y1 = (int)height;

ras­ter­iz­er_­glob­als.frame_bounds.x0 = x_off;

ras­ter­iz­er_­glob­als.frame_bounds.y0 = y_off;

ras­ter­iz­er_­glob­als.frame_bounds.x1 = (int)width - x_off;

ras­ter­iz­er_­glob­als.frame_bounds.y1 = (int)height - y_off;

bool ras­ter­iz­er_de­vice_ini­tial­ize()

D3DPRESENT_PARAMETERS PresentParams = {0};

PresentParams. BackBufferWidth = ras­ter­iz­er_­glob­als.screen_bounds.x1 - ras­ter­iz­er_­glob­als.screen_bounds.x0;

PresentParams.BackBufferHeight = ras­ter­iz­er_­glob­als.screen_bounds.y1 - ras­ter­iz­er_­glob­als.screen_bounds.y1;

PresentParams.BackBufferFormat = D3DFMT_A8R8G8B8;

PresentParams.EnableAutoDepthStencil = TRUE;

PresentParams.AutoDepthStencilFormat = D3DFMT_D24S8;

PresentParams.Flags = D3DPRESENTFLAG_LOCKABLE_BACKBUFFER;

PresentParams.FullScreen_RefreshRateInHz = g_re­fresh_rate_hz;

PresentParams.FullScreen_PresentationInterval = D3DPRESENT_INTERVAL_IMMEDIATE;

// Check if wide screen mode is en­abled.

if (g_widescreen_enabled != 0)

PresentParams.Flags |= D3DPRESENTFLAG_WIDESCREEN;

// Check if the video mode sup­ports pro­gres­sive scan.

if (g_progressive_scan_enabled != 0)

PresentParams.Flags |= D3DPRESENTFLAG_PROGRESSIVE;

// Check the res­o­lu­tion width to see if 1080i is en­abled.

if (rasterizer_globals.screen_bounds.x1 == 1920)

PresentParams.Flags &= ~D3DPRESENTFLAG_PROGRESSIVE;

PresentParams.Flags |= D3DPRESENTFLAG_INTERLACED;

g_pDi­rec­t3D->Cre­at­eDe­vice(0, D3DDEVTYPE_HAL, NULL, D3DCREATE_HARDWARE_VERTEXPROCESSING, &PresentParams, &g_pD3DDevice);

// Check the res­o­lu­tion width to see if 1080i is en­abled.

Booting up the game with these changes gives some less than pleas­ing re­sults. Looking at the main menu the first thing we can see is the blue fil­ter is now gone, and there’s some re­peat­ing line pat­tern strewn across the screen. Looking a bit closer and we can see part of the wa­ter geom­e­try is also cut off, sus­pi­ciously at where the old 640 width would be com­pared to the new width of 720.

The Xbox uses a uni­fied mem­ory ar­chi­tec­ture mean­ing the CPU and GPU share the same RAM. Unlike a PC there’s no con­cept of cre­at­ing a D3D al­lo­ca­tion in VRAM and hav­ing the GPU man­age it. On Xbox the CPU can cre­ate an al­lo­ca­tion for tex­tures, ren­der tar­gets, ver­tex buffers, etc, and pass the al­lo­ca­tion ad­dress di­rectly to the GPU. This gives de­vel­op­ers the abil­ity to al­lo­cate one buffer and have mul­ti­ple re­source “views” that uti­lize the mem­ory. Consider the fol­low­ing code which shows how to cre­ate a ren­der tar­get let­ting D3D do all the work and how to cre­ate a ren­der tar­get by hand:

// How to cre­ate a ren­der tar­get with D3D:

IDirect3DSurface8* pRen­der­Tar­get = NULL;

g_pD3D­De­vice->Cre­ateRen­der­Tar­get(/* width */ 1024, /* height */ 1024, /* for­mat */ D3DFMT_A8R8G8B8, NULL, FALSE, &pRenderTarget);

// How to cre­ate a ren­der tar­get by hand:

// Allocate and ini­tial­ize the tex­ture header.

IDirect3DSurface8* pRen­der­Tar­get = (IDirect3DSurface8*)malloc(sizeof(IDirect3DSurface8));

DWORD tex­ture­Size = XGSetTextureHeader(/* width */ 1024, /* height */ 1024, /* lev­els */ 0, 0, /* for­mat */ D3DFMT_A8R8G8B8, 0, pRen­der­Tar­get, 0, 0);

// Allocate mem­ory for the pixel buffer.

void* pSur­face­Buffer = D3D_AllocContiguousMemory(/* size */ tex­ture­Size, /* align­ment */ D3DSURFACE_ALIGNMENT);

pRen­der­Tar­get->Reg­is­ter(pSur­face­Buffer);

// How to cre­ate a ren­der tar­get with D3D:// How to cre­ate a ren­der tar­get by hand:

While the lat­ter looks more messy it pro­vides greater con­trol to the de­vel­oper and is some­thing Halo 2 makes great use of to con­serve mem­ory for all the ren­der tar­gets it uses. In to­tal Halo 2 has ap­prox­i­mately 25 dif­fer­ent ren­der tar­gets it uses but there’s only 4-5 unique buffers al­lo­cated for them which saves a lot of mem­ory. So what does this have to do with the is­sues we saw in the main menu? Well if Halo 2 is cre­at­ing ren­der tar­gets by hand it’ll need to en­code the width and height of the sur­face into the header of the ren­der tar­get struc­ture. If it’s hard coded to use 640×480 res­o­lu­tion it would cause is­sues that could re­sult in cut off im­ages or re­peat­ing line pat­terns as the pitch of the sur­face would not match the pitch of the back buffer. Essentially, there’s two dif­fer­ent “views” for the same mem­ory but the views see the mem­ory as be­ing of dif­fer­ent widths which re­sults in mis­placed pix­els when span­ning each scan line.

Looking around the D3D/raster ini­tial­iza­tion code I found a func­tion I called ras­ter­iz­er_pri­ma­ry_­tar­get­s_ini­tial­ize that does ex­actly this. It takes the back, front, and depth buffers cre­ated by D3D and cre­ates ad­di­tional ren­der tar­gets and tex­ture views from them, us­ing hard coded di­men­sions of 640×480. Here is the C rep­re­sen­ta­tion of the dis­as­sem­bly:

bool ras­ter­iz­er_pri­ma­ry_­tar­get­s_ini­tial­ize()

// Get the back buffer, front buffer, and depth buffer sur­faces.

glob­al_d3d_de­vice->Get­Back­Buffer(0, D3DBACKBUFFER_TYPE_MONO, &global_d3d_surface_render_primary[0]);

glob­al_d3d_de­vice->Get­Back­Buffer(-1, D3DBACKBUFFER_TYPE_MONO, &global_d3d_surface_render_primary[1]);

glob­al_d3d_de­vice->Get­Depth­S­ten­cil­Sur­face(&glob­al_d3d_­sur­face_ren­der_pri­ma­ry_z);

glob­al_d3d_­tex­ture_ren­der_pri­mary[0] = (IDirect3DTexture8*)malloc(sizeof(IDirect3DTexture8));

glob­al_d3d_­tex­ture_ren­der_pri­mary[1] = (IDirect3DTexture8*)malloc(sizeof(IDirect3DTexture8));

// Setup tex­ture views for back/​front buffers.

for (int i = 0; i < 2; i++)

XGSetTextureHeader(640, 480, 1, 0, D3DFMT_LIN_A8R8G8B8, 0,

glob­al_d3d_­tex­ture_ren­der_pri­mary[i], glob­al_d3d_­sur­face_ren­der_pri­mary[i]->Data, 0);

// Create a ren­der tar­get sur­face for the depth buffer that matches the size and for­mat of the back buffer.

glob­al_d3d_­sur­face_z_as_­tar­get = (IDirect3DSurface8*)malloc(sizeof(IDirect3DSurface8));

mem­cpy(glob­al_d3d_­sur­face_z_as_­tar­get, glob­al_d3d_­sur­face_ren­der_pri­mary, sizeof(IDi­rec­t3D­Sur­face8));

glob­al_d3d_­sur­face_z_as_­tar­get->Data = glob­al_d3d_­sur­face_ren­der_pri­ma­ry_z->Data;

// Create two tex­tures for the depth buffer, one in ARGB for­mat and one in ABGR for­mat.

glob­al_d3d_­tex­ture_z_as_­tar­get[0] = (IDirect3DTexture8*)malloc(sizeof(IDirect3DTexture8));

XGSetTextureHeader(640, 480, 1, 0, D3DFMT_LIN_A8R8G8B8, 0,

glob­al_d3d_­tex­ture_z_as_­tar­get[0], glob­al_d3d_­sur­face_ren­der_pri­ma­ry_z->Data, 0);

In October 2022 a bird with the code name B6 set a new world record that few peo­ple out­side the field of or­nithol­ogy no­ticed. Over the course of 11 days, B6, a young Bar-tailed Godwit, flew from its hatch­ing ground in Alaska to its win­ter­ing ground in Tasmania, cov­er­ing 8,425 miles with­out tak­ing a sin­gle break. For com­par­i­son, there is only one com­mer­cial air­craft that can fly that far non­stop, a Boeing 777 with a 213-foot wingspan and one of the most pow­er­ful jet en­gines in the world. During its jour­ney, B6—an an­i­mal that could perch com­fort­ably on your shoul­der—did not land, did not eat, did not drink and did not stop flap­ping, sus­tain­ing an av­er­age ground speed of 30 miles per hour 24 hours a day as it winged its way to the other end of the world.

Many fac­tors con­tributed to this as­ton­ish­ing feat of ath­leti­cism—mus­cle power, a high meta­bolic rate and a phys­i­o­log­i­cal tol­er­ance for el­e­vated cor­ti­sol lev­els, among other things. B6’s odyssey is also a tri­umph of the re­mark­able me­chan­i­cal prop­er­ties of some of the most eas­ily rec­og­nized yet enig­matic struc­tures in the bi­o­log­i­cal world: feath­ers. Feathers kept B6 warm overnight while it flew above the Pacific Ocean. Feathers re­pelled rain along the way. Feathers formed the flight sur­faces of the wings that kept B6 aloft and drove the bird for­ward for nearly 250 hours with­out fail­ing.

One might ex­pect that, con­sid­er­ing all the time hu­mans have spent ad­mir­ing, us­ing and study­ing feath­ers, we would know all their tricks by now. Yet in­sights into these mar­velous struc­tures con­tinue to emerge. Over the past decade other re­searchers and I have been tak­ing a fresh look at feath­ers. Collectively we have made sur­pris­ing new dis­cov­er­ies about al­most every as­pect of their bi­ol­ogy, from their evo­lu­tion­ary ori­gins to their growth, de­vel­op­ment and aero­dy­nam­ics.

If you’re en­joy­ing this ar­ti­cle, con­sider sup­port­ing our award-win­ning jour­nal­ism by sub­scrib­ing. By pur­chas­ing a sub­scrip­tion you are help­ing to en­sure the fu­ture of im­pact­ful sto­ries about the dis­cov­er­ies and ideas shap­ing our world to­day.

Among the crea­tures we share the planet with to­day, only birds have feath­ers. It makes sense, then, that for cen­turies sci­en­tists con­sid­ered feath­ers a unique fea­ture of birds. But start­ing in the 1990s, a se­ries of bomb­shell fos­sil finds es­tab­lished that feath­ers were wide­spread among sev­eral lin­eages of the bipedal, car­niv­o­rous di­nosaurs known as theropods and that birds had in­her­ited these struc­tures from their thero­pod an­ces­tors. The dis­cov­ery of feath­ered non­bird di­nosaurs sent re­searchers scram­bling to un­der­stand the ori­gin and evo­lu­tion of feath­ers, es­pe­cially their role in the dawn of flight. We now know many di­nosaurs had feath­ers, and protofeath­ers prob­a­bly go all the way back to the com­mon an­ces­tor of di­nosaurs and their fly­ing rep­tile cousins, the pterosaurs. Bristles, fuzzy cov­er­ings, and other rel­a­tively sim­ple feath­er­like struc­tures prob­a­bly dec­o­rated a wide ar­ray of di­nosaurs—many more than we have been lucky enough to find pre­served as fos­sils.

The feath­ers on non­bird di­nosaurs were not lim­ited to bris­tles and fuzz, how­ever. The flat, broad, flight-en­abling feath­ers we see across most of the wings and much of the body sur­face of liv­ing birds are called pen­na­ceous feath­ers. (Fun fact: these are the feath­ers peo­ple used to make into quills for writ­ing, hence the word “pen.”) It turns out that these feath­ers, too, ap­peared be­fore birds. In fact, there is an en­tire group of di­nosaurs com­pris­ing birds as well as species such as Velociraptor that takes its name from these very feath­ers: the penna­raptoran clade. Fossils of early pen­nara­p­torans show that they had feath­ery cov­er­ings that would have looked es­sen­tially mod­ern at a quick glance.

The flight ca­pac­ity of these early pen­nara­p­torans has been hotly con­tested. Some species were clearly not fliers, given the small size of their “wings” rel­a­tive to their large bod­ies. For those an­i­mals, pen­na­ceous feath­ers were prob­a­bly dis­play pieces. But other pen­nara­p­torans, such as the small, four-winged, for­est-dwelling Microraptor, are trick­ier to in­ter­pret. Many of the ar­gu­ments about whether this crea­ture could fly have cen­tered on some­thing called vane asym­me­try. The two flat “blades” of a feather on ei­ther side of the main shaft are called vanes. In liv­ing birds that fly, the feath­ers that arise from the hand, known as the pri­maries, have asym­met­ri­cal vanes: the lead­ing vane is nar­rower than the trail­ing one. It stood to rea­son that vane asym­me­try was im­por­tant for flight. And be­cause fos­sils of Microraptor and its kin show asym­met­ri­cal feath­ers, some re­searchers ar­gued, these an­i­mals must have been able to fly.

Recent work by flight bio­me­chan­ics ex­perts, in­clud­ing me, has over­turned this re­ceived wis­dom about feather vane asym­me­try. Our re­search shows that feather shape is largely op­ti­mized to al­low the feather to twist and bend in so­phis­ti­cated ways that greatly en­hance flight per­for­mance. Merely be­ing anatom­i­cally asym­met­ri­cal does­n’t mean much. What mat­ters is that the feather is aero­dy­nam­i­cally asym­met­ri­cal, and for this to be the case, the vane asym­me­try must be at least three to one—that is, the trail­ing blade needs to be three times wider than the lead­ing one. Below this ra­tio, the feather twists in a desta­bi­liz­ing rather than sta­bi­liz­ing way dur­ing flight.

Early pen­nara­p­torans such as Microraptor did­n’t have aero­dy­nam­i­cally asym­met­ri­cal feath­ers. But that does­n’t mean they could­n’t fly. The ten­dency to twist (whether in a sta­bi­liz­ing or a desta­bi­liz­ing fash­ion) is only rel­e­vant if the feath­ers are sep­a­rated enough to do so. Keeping feath­ers in a wing tip tight and over­lap­ping makes them sta­ble, even if they’re not asym­met­ri­cal. Asymmetry mat­ters only if the flier spreads its pri­maries apart in flight like many mod­ern rap­tors do—a fea­ture called slot­ting. So Microraptor and its kin could prob­a­bly use flap­ping flight, but their wing shape was nec­es­sar­ily dif­fer­ent from that of to­day’s for­est-dwelling birds of prey. Specifically, Microraptor had rel­a­tively long, nar­row wings with tight, un­slot­ted wing tips—anatom­i­cally dis­tinct from the wings of Cooper’s Hawks and other mod­ern-day for­est hawks but aero­dy­nam­i­cally sim­i­lar.

After con­sid­er­ing these find­ings on vane asym­me­try, as well as new data on flight mus­cles in near-bird di­nosaurs, a group of re­searchers (of which I was the se­nior bio­physi­cist) led by Michael Pittman of the Chinese University of Hong Kong re­cently con­cluded that pow­ered flight—that is, flap­ping flight rather than glid­ing flight—prob­a­bly evolved mul­ti­ple times in di­nosaurs, with just one of those lin­eages sur­viv­ing to the pre­sent in the form of birds. Yet only in birds did flight feath­ers at­tain the de­gree of shape-shift­ing we see to­day. That abil­ity of feath­ers to twist in just the right way is what en­abled slot­ting, which makes the wing much more ef­fi­cient at low flight speeds. In essence, a slot­ted wing be­haves as if it is longer and nar­rower than it is anatom­i­cally. Slotting also makes the wing tip very re­sis­tant to stall, whereby the air­flow sep­a­rates from the wing, caus­ing a pre­cip­i­tous loss of the lift that keeps the bird in the air. It’s a vi­tal adap­ta­tion that un­der­pins an ar­ray of aer­ial ac­ro­bat­ics.

Birds typ­i­cally need long, nar­row wings to soar ef­fi­ciently—seabirds such as al­ba­trosses and pe­trels are per­fect ex­am­ples. The ad­vent of wing-tip slots made it pos­si­ble to soar with broader wings, paving the way for evo­lu­tion of a di­ver­sity of broad-winged soar­ers, in­clud­ing vul­tures and hawks. The aero­dy­namic ad­van­tages of slot­ting also per­mit the ex­plo­sive flight per­for­mance of sprint­ers such as grouses, which spend most of their time on the ground but burst into flight for a short dis­tance when star­tled. And wing-tip slots pro­vide much greater ma­neu­ver­abil­ity for a wide ar­ray of birds that live in forests and other clut­tered en­vi­ron­ments, from song­birds to tou­cans. In fact, the ma­neu­ver­abil­ity made pos­si­ble by slot­ted wings might have helped birds com­pete with pterosaurs and ul­ti­mately sur­vive the end-Cre­ta­ceous ex­tinc­tion.

The pen­na­ceous feath­ers we as­so­ci­ate with flight aren’t the only type of feather birds pos­sess. Feathers in dif­fer­ent re­gions of the body vary in size, shape and func­tion. You can think of feather form as a spec­trum, with the large, rel­a­tively stiff flight feath­ers of the wing and tail at one end and the short, fluffy down feath­ers that sit close to the body at the other. All of them have a cen­tral shaft and softer “barbs” that branch out from the shaft. In flight feath­ers, the barbs in­ter­lock like Velcro teeth to form the smooth, wind­proof sur­face of the vanes. In down feath­ers, the barbs are loosely struc­tured and fluffy to trap heat. Many of the other kinds of feath­ers com­bine as­pects of these two types. The con­tour feath­ers that stream­line a bird’s body, for ex­am­ple, have vaned tips like flight feath­ers and non­in­ter­lock­ing barbs like down ones. The bris­tle feath­ers that typ­i­cally oc­cur on the face and may serve pro­tec­tive and sen­sory pur­poses meld the flight feath­ers’ stiff shafts (called rachises) with the down feath­ers’ fluffy base.

In re­cent years re­searchers have be­gun to piece to­gether the in­tri­cate process by which feath­ers de­velop. Like scales, spines and hairs, feath­ers are skin ap­pendages. Scientists have known for a while that they arise from struc­tures in the skin. But how can an an­i­mal pro­duce feath­ers with dif­fer­ent anatomies across its body?

My col­leagues and I, led by Cheng-Ming Chuong of the University of Southern California, re­lated the de­vel­op­men­tal bi­ol­ogy of var­i­ous kinds of pen­na­ceous feath­ers to their me­chan­i­cal prop­er­ties. These feath­ers be­gin as a tube that es­sen­tially un­zips along its length, form­ing the two vanes. Several genes and mol­e­cules in­ter­act with one an­other and with the en­vi­ron­ment to de­ter­mine the amount of in­ter­lock­ing in the barbs that make up the vanes, the size and shape of the rachises, and whether the shaft is filled with a “foam” that makes it stiff rel­a­tive to its weight. We found that dif­fer­ent feather types have vary­ing spe­cial­iza­tions in their over­all stiff­ness, their ten­dency to twist, and the dis­tri­b­u­tion of the foam in the shaft. These vari­a­tions de­pend to some ex­tent on the work of dif­fer­ent genes, but most of the dif­fer­en­ti­a­tion is the re­sult of changes in how the genes are reg­u­lated—that is, when they are turned on or off or how ac­tive they are dur­ing feather de­vel­op­ment.

Scientists have also shown a re­cent surge of in­ter­est in an­other cat­e­gory of feath­ers: dis­play feath­ers, the showy feath­ers that help to at­tract mates. Display feath­ers may daz­zle an ob­server with their col­ors (think of a hum­ming­bird’s glit­ter­ing throat), or they may at­tain eye-catch­ing pro­por­tions, like the feath­ers that make up a pea­cock’s crest and train. The con­ven­tional wis­dom about dis­play feath­ers holds that they are strictly prod­ucts of sex­ual se­lec­tion, in which mate choice dri­ves the evo­lu­tion of a trait. These days, how­ever, re­searchers around the globe, me in­cluded, are com­ing to see dis­play feath­ers not as ex­clu­sively sex­u­ally se­lected traits with no me­chan­i­cal prop­er­ties of in­ter­est but in­stead as com­plex com­pro­mises be­tween the pres­sures of so­cial bi­ol­ogy and mechanobi­ol­ogy.

To wit: long dis­play feath­ers don’t grow just any­where on the body. They most of­ten oc­cur on the lower back and tail, where they in­ter­fere com­par­a­tively lit­tle with flight per­for­mance. Take, for ex­am­ple, the Resplendent Quetzal, a small, col­or­ful bird na­tive to the cloud forests of Mexico and Central America with tail feath­ers that can grow up to three feet long on males dur­ing the breed­ing sea­son. The tail stream­ers might not be shaped solely by sex­ual se­lec­tion. Evidence in­di­cates that the stream­ers of some birds pro­duce at least a lit­tle aero­dy­namic force, enough to sup­port much of their added weight. The quet­za­l’s stream­ers, for their part, lost their tight in­ter­lock­ing struc­ture, mak­ing the vanes a pen­na­ceous-downy hy­brid that lets much of the air­flow pass through with­out pro­duc­ing much lift. This arrange­ment is most likely an adap­ta­tion to pre­vent these feath­ers from be­ing highly desta­bi­liz­ing. These flashy feath­ers still in­crease the cost of fly­ing be­cause they add drag, but that cost may well be less than has been as­sumed.

The mi­crostruc­ture of dis­play feath­ers, es­pe­cially tail stream­ers, may also be more finely tuned than pre­vi­ously thought. Feather struc­ture pro­vides a bal­ance of stiff­ness, weight and shape. The feath­ers must hold their shape well enough, even at ex­treme lengths, to be ef­fec­tive sig­nals. But they can­not be so stiff as to desta­bi­lize the bird dur­ing gusty winds or tight ma­neu­vers. There’s a par­tic­u­lar range of flex­i­bil­ity that shows off the feather to best ef­fect while min­i­miz­ing detri­men­tal im­pacts on flight per­for­mance.

One of the as­pects of feath­ers that has long fas­ci­nated me is their adapt­abil­ity. Under vary­ing con­di­tions and evo­lu­tion­ary pres­sures, they can be­come spe­cial­ized for every­thing from speed and ma­neu­ver­abil­ity to in­su­la­tion or dis­play. Some of the most fas­ci­nat­ing adap­ta­tions can be found in owls.

Facial disks are an es­pe­cially con­spic­u­ous fea­ture of owls. These broad, semi­cir­cu­lar fans of feath­ers around the eyes and ears give owls their dis­tinc­tive ap­pear­ance. The skull of an owl is ac­tu­ally quite long and nar­row, but the feath­ers en­velop­ing it com­pletely change the con­tours of the an­i­mal. These fa­cial disks are not just for looks. They do a re­mark­ably good job of fun­nel­ing sound to the owl’s ears. The disks, along with ver­ti­cally off­set ears and ex­cep­tion­ally sen­si­tive mid­dle and in­ner ear struc­tures, make owls so good at de­ter­min­ing the ori­gin of a sound that they can zero in on prey with­out see­ing it at all (they still use vi­sion to make the fi­nal cap­ture, though).

I have worked with quite a few owls over the years, par­tic­u­larly in­di­vid­u­als be­ing re­ha­bil­i­tated af­ter in­jury. One such owl could­n’t be re­leased be­cause a car strike had left him com­pletely blind. Yet if some­one tossed food onto one of his perches, the gen­tle thud of it land­ing was enough for him to pounce on it per­fectly. (Readers may also find so­lace in know­ing that he still flew, hav­ing mem­o­rized his en­clo­sure, and was reg­u­larly taken around for walks and neck scratches.)

Still, that ex­cep­tional sense of hear­ing would­n’t get owls very far with­out some ad­di­tional feather adap­ta­tions. Other noc­tur­nal crea­tures can also hear very well, and an owl whose feath­ers were rustling in flight would be hard-pressed to get close to its vig­i­lant prey. Furthermore, owls might not hear qui­etly creep­ing prey if their own feather sounds cov­ered the faint noises of their tar­gets. Owls solved both prob­lems by evolv­ing feather traits that make them in­audi­ble dur­ing flight.

It is hard to ap­pre­ci­ate just how quiet owls are. Even ul­tra­sen­si­tive mi­cro­phones, if prop­erly cal­i­brated, aimed ex­actly right and set to max­i­mum sen­si­tiv­ity in a silent space, can just barely pick up sounds from a fly­ing owl … some­times. For all prac­ti­cal pur­poses, owls are silent. They are so eerily noise­less that even if they fly over your head close enough for you to feel their wake, you will still hear ab­solutely noth­ing. In a dark space, they are es­sen­tially un­de­tectable. All the owl wing sounds you hear in the Harry Potter movies and other films? Those are added in.

Owls achieve this stealth with a few dif­fer­ent feather adap­ta­tions. To start, their feath­ers have a “velvety” sur­face that si­lences them when they move against one an­other. More im­por­tant, the feath­ers on the lead­ing edge of an owl’s wing have a set of comb­like struc­tures, whereas those on the trail­ing edge have fluffy fringes. The lead­ing-edge comb stirs the air in a spe­cific way called mi­cro vor­tic­ity. These tiny, swirling streams of air cause the main flow to stick to the wing. In aero­dy­namic speak, we say the combs “inject vor­tic­ity into the bound­ary layer.” When this mod­i­fied flow then passes through the trail­ing-edge fringes, the net re­sult is a wake that con­tains no co­her­ent waves of lin­ear pres­sure and there­fore no sound. Put an­other way, there are no vi­bra­tions from the in­ter­ac­tions be­tween feath­ers and the air ca­pa­ble of pro­duc­ing sound.

These spe­cial­iza­tions have deep roots. Modern-day owls be­long to one of two groups: the ty­tonids (represented by Barn Owls and Bay Owls) and the st­rigids (all other liv­ing owls). Their last com­mon an­ces­tor ex­isted at least 50 mil­lion years ago. Because owls in both groups ex­hibit silent flight, this trait prob­a­bly dates to their com­mon an­ces­tor. In other words, owls have been sur­rep­ti­tiously cours­ing the night skies for more than 50 mil­lion years.

Not sur­pris­ingly, some of the most ex­treme feather adap­ta­tions are found in birds with the most ex­treme eco­log­i­cal spe­cial­iza­tions. One way feath­ers can adapt to a par­tic­u­lar way of life is by in­creas­ing or de­creas­ing in stiff­ness. Coincidentally, the stiffest feath­ers are found in two groups of birds that are oth­er­wise as dif­fer­ent as can be: hum­ming­birds and pen­guins.

Hummingbirds have ul­tra­stiff feath­ers as an adap­ta­tion to the ex­cep­tion­ally high flap­ping fre­quen­cies and un­usual flap­ping stroke they use to hover in front of flow­ers while sip­ping nec­tar. Unlike most birds, hum­ming­birds can get a sub­stan­tial amount of weight sup­port and thrust from their up­stroke, not just their down­stroke. They do this by ro­tat­ing their shoul­der to flip the wing over com­pletely. The wing needs to be very stiff for this method to work. Reinforcements in the bones of the hum­ming­bird wing pro­vide some of this rigid­ity; feath­ers with ex­tremely firm rachises pro­vide the rest.

The flight­less pen­guins, in con­trast, have adapted to life in the wa­ter and on land. They pos­sess some of the most spe­cial­ized plumage of all, hav­ing con­verted their en­tire body cov­er­ing into a densely packed mo­saic of tiny feath­ers. These feath­ers are in­di­vid­u­ally quite stiff, and to­gether they form a tex­tured sur­face over the wings and body that reg­u­lates the bound­ary layer of wa­ter against them while the pen­guin is swim­ming. In essence, they use a rough coat of feath­ers to catch and hold a smooth jacket of wa­ter. The net ef­fect is a re­duc­tion in drag and there­fore a lower en­er­getic cost of swim­ming. The dense feath­ers also trap just enough air to pro­vide some in­su­la­tion with­out mak­ing the pen­guin buoy­ant, sup­ple­ment­ing the fat layer that helps to keep the bird warm.

In the ab­sence of any con­straints posed by flight, pen­guins jet­ti­soned the more typ­i­cal feather ac­cou­trements of their an­ces­tors in fa­vor of a novel suit of drag-­­reducing, min­i­mum-buoy­ancy feath­ers. These feath­ers are a key part of the pack­age of adap­ta­tions that have made pen­guins the undis­puted div­ing cham­pi­ons of the avian world, ca­pa­ble of reach­ing depths of more than 1,600 feet in search of krill, fish, and other aquatic prey.

Feathers are a fan­tas­tic model sys­tem for un­der­stand­ing how com­plex struc­tures evolve and how anatomy and be­hav­ior in­flu­ence each other over time. It’s no won­der that the ap­plied sci­ence sec­tor has taken note of feath­ers’ many bril­liant fea­tures. Already they have led to suc­cess­ful tech­no­log­i­cal in­no­va­tions. The Velcro-like mech­a­nism that con­nects the barbs of pen­na­ceous feath­ers is the ba­sis for an ad­vanced tem­po­rary fas­ten­ing sys­tem. The si­lenc­ing fringes of owl feath­ers have in­spired ven­ti­la­tion-qui­et­ing sys­tems. The sur­face tex­ture and bound­ary-layer-con­trol prin­ci­ples of pen­guin feath­ers have made their way into ro­bot­ics, mostly in pro­to­types.

No doubt feath­ers will give rise to more clever in­ven­tions in the fu­ture. We have only to let our cre­ativ­ity take flight.

To see all avail­able qual­i­fiers, see our doc­u­men­ta­tion.

We read every piece of feed­back, and take your in­put very se­ri­ously.

To see all avail­able qual­i­fiers, see our doc­u­men­ta­tion.

We read every piece of feed­back, and take your in­put very se­ri­ously.

You signed in with an­other tab or win­dow. Reload to re­fresh your ses­sion.

You signed out in an­other tab or win­dow. Reload to re­fresh your ses­sion.

You switched ac­counts on an­other tab or win­dow. Reload to re­fresh your ses­sion.

OnOn the af­ter­noon of March 11th, 2011, Mitsuyoshi Hirai, the chief en­gi­neer of the ca­ble main­te­nance ship Ocean Link, was sit­ting in his cabin 20 miles off Japan’s east­ern coast, com­plet­ing the pa­per­work that comes at the end of every re­pair. Two weeks ear­lier, some­thing — you rarely knew what — dam­aged the 13,000-mile fiber op­tic ca­ble con­nect­ing Kitaibaraki, Japan, and Point Arena, California. Alarms went off; calls were made; and the next day, Hirai was sail­ing out of the port in Yokohama to fix it.

A cam­era mounted on the KDDI Ocean Link on March 11th, 2011.

The re­pair was now nearly done. All that re­mained was to re­bury the ca­ble on the seafloor, which they were do­ing us­ing a bull­dozer-sized re­motely op­er­ated sub­mersible named Marcas — and, of course, the pa­per­work.

Suddenly, the ship be­gan to shud­der. Hirai got to his feet, found he could barely stand, and stag­gered out of his cabin, grasp­ing the handrail as he pulled him­self up the nar­row stair­way to the bridge. “Engine trou­ble?” Hirai asked the cap­tain, who’d al­ready checked and replied that every­thing seemed nor­mal. The ship con­tin­ued to trem­ble. Looking out from the bridge, the sea ap­peared to be boil­ing.

A sketch of the Ocean Link in port in Yokohama tran­si­tions into a video of the ship. A bird flies over­head and waves lap at its hull.

They turned on the tele­vi­sion. An emer­gency alert showed that an earth­quake had struck 130 miles north­east of their lo­ca­tion. The shak­ing fi­nally stopped, and in the si­lence, Hirai’s mind leapt to what would come next: a tsunami.

Hirai feared these waves more than most peo­ple. He had grown up hear­ing the story of how one af­ter­noon in 1923, his aunt felt the ground shake, swept up her two-year-old brother, and sprinted up­hill to the ceme­tery, nar­rowly es­cap­ing floods and fires that killed over 100,000 peo­ple. That child be­came Hirai’s fa­ther, so he owed his ex­is­tence to his aun­t’s quick think­ing. Now, he found him­self in the same po­si­tion. He knew tsunamis be­come dan­ger­ous when all the wa­ter dis­placed by the quake reaches shal­low wa­ter and slows and grows taller. The Ocean Link, float­ing in less than 500 feet of wa­ter, was too shal­low for com­fort.

Mitsuyoshi Hirai, the for­mer chief en­gi­neer of the Ocean Link.

Photo by Go Takayama for The Verge

In the fam­ily tree of pro­fes­sions, sub­ma­rine ca­ble work oc­cu­pies a lonely branch some­where be­tween heavy con­struc­tion and neu­ro­surgery. It’s pre­ci­sion en­gi­neer­ing on a shift­ing sea us­ing heavy metal hooks and high-ten­sion lines that, if they snap, can cut a per­son in half. In Hirai’s three decades with Kokusai Cable Ship Company (KCS), he had learned that every step must be fol­lowed, no mat­ter how chaotic the sit­u­a­tion. Above all else, he of­ten said, “you must al­ways be cool.”

Across Ocean Link’s 400-foot deck, the ship’s 50 crew mem­bers were emerg­ing from their cab­ins and work­sta­tions, try­ing to fig­ure out what had just oc­curred. Over the in­ter­com, the cap­tain an­nounced that there had been an earth­quake, a tsunami was com­ing, and the crew should ready the ship to evac­u­ate to deeper wa­ter. The crew fanned out to check fuel tanks and lash down ma­chin­ery. Inside a dark­ened, mon­i­tor-filled ship­ping con­tainer on the star­board deck, the sub­mersible’s pi­lot steered Marcas back to­ward the ship as fast as the bulky ro­bot’s pro­pellers could carry it. Minutes later, the sub­mersible was hoisted aboard and the Ocean Link was un­der­way.

Controls on the bridge of the Ocean Link.

Photo by Go Takayama for The Verge

View from the bridge of the Ocean Link.

Photo by Go Takayama for The Verge

Photo by Go Takayama for The Verge

Photo by Go Takayama for The Verge

The tsunami passed un­der them im­per­cep­ti­bly on their way out to sea, and when they came to a stop three hours later, the tele­vi­sion was show­ing the first im­ages of de­struc­tion. Members of the crew who weren’t work­ing gath­ered on the bridge to watch the news, which con­tin­ued to dis­play a tsunami warn­ing, a map of Japan with its east­ern seaboard glow­ing red. They took turns try­ing to reach loved ones us­ing the ship’s satel­lite phone, but no calls went through.

As night fell, pe­ri­odic af­ter­shocks thumped against the hull. Hirai thought about his wife, who was work­ing at a de­part­ment store in Yokohama near the Ocean Link’s port; his son, a ju­nior in high school at the time; and his par­ents, whom the fam­ily lived with in his home­town of Yokosuka — none of whom he’d been able to reach. Everyone had some­one they were wor­ried about.

But Hirai also be­gan to think about the work he knew lay ahead. The Ocean Link was one of a small num­ber of ships that main­tain the sub­sea ca­bles that carry 99 per­cent of the world’s data. Positioned in strate­gic lo­ca­tions around the planet, these ships stand ready to sail out and fix faults the mo­ment they are de­tected, and most of the time, they are more than equal to the task. But earth­quakes, Hirai knew from ex­pe­ri­ence, were dif­fer­ent. They did­n’t just break one ca­ble — they broke many, and badly. If what he feared had hap­pened, Japan risked be­ing cut off from the world in its mo­ment of need.

Sure enough, that night, a call came from head­quar­ters con­firm­ing the Ocean Link was safe and di­rect­ing them to re­main at sea un­til fur­ther no­tice, fol­lowed by mes­sages an­nounc­ing ca­ble fail­ure af­ter ca­ble fail­ure, in­clud­ing the one they had just fin­ished re­pair­ing.

Fumihide Kobayashi stand­ing in front of the sub­mersible Marcas.

Photo by Go Takayama for The Verge

Cable in­dus­try pro­fes­sion­als tend to be prag­matic peo­ple, pre­oc­cu­pied with the ma­te­r­ial re­al­i­ties of work­ing planet-scale con­struc­tion. But in con­ver­sa­tions about land­ing high-band­width ca­bles in dig­i­tally ne­glected re­gions or putting mil­lions of peo­ple back in con­tact with every fiber strand melted to­gether, they of­ten hint at a sense of larger pur­pose, an aware­ness that they are per­form­ing a func­tion vi­tal to a world that, if they do their jobs well, will con­tinue to be un­aware of their ser­vice.

For the Ocean Link crew, this aware­ness was bound up in a still un­fold­ing na­tional tragedy. They knew that when­ever they re­turned to land, they would have to care for their loved ones quickly, be­cause they would soon be go­ing back out to sea. For how long, no one knew.

TheThe world’s emails, TikToks, clas­si­fied memos, bank trans­fers, satel­lite sur­veil­lance, and FaceTime calls travel on ca­bles that are about as thin as a gar­den hose. There are about 800,000 miles of these skinny tubes criss­cross­ing the Earth’s oceans, rep­re­sent­ing nearly 600 dif­fer­ent sys­tems, ac­cord­ing to the in­dus­try track­ing or­ga­ni­za­tion TeleGeography. The ca­bles are buried near shore, but for the vast ma­jor­ity of their length, they just sit amid the gray ooze and alien crea­tures of the ocean floor, the hair-thin strands of glass at their cen­ter glow­ing with lasers en­cod­ing the world’s data.

If, hy­po­thet­i­cally, all these ca­bles were to si­mul­ta­ne­ously break, mod­ern civ­i­liza­tion would cease to func­tion. The fi­nan­cial sys­tem would im­me­di­ately freeze. Currency trad­ing would stop; stock ex­changes would close. Banks and gov­ern­ments would be un­able to move funds be­tween coun­tries be­cause the Swift and US in­ter­bank sys­tems both rely on sub­ma­rine ca­bles to set­tle over $10 tril­lion in trans­ac­tions each day. In large swaths of the world, peo­ple would dis­cover their credit cards no longer worked and ATMs would dis­pense no cash. As US Federal Reserve staff di­rec­tor Steve Malphrus said at a 2009 ca­ble se­cu­rity con­fer­ence, “When com­mu­ni­ca­tions net­works go down, the fi­nan­cial ser­vices sec­tor does not grind to a halt. It snaps to a halt.”

A map of the world show­ing the dozens of fi­bre op­tic ca­ble sys­tems which stretch across the oceans, con­nect­ing con­ti­nents and is­land chains. Some of these ca­bles are ex­tremely long. The map an­i­mates to show the ca­bles laid down be­tween 1989 and the pre­sent, with planned ca­bles up to 2027 also dis­played.

Active and planned fiber op­tic ca­ble sys­tems

Corporations would lose the abil­ity to co­or­di­nate over­seas man­u­fac­tur­ing and lo­gis­tics. Seemingly lo­cal in­sti­tu­tions would be par­a­lyzed as out­sourced ac­count­ing, per­son­nel, and cus­tomer ser­vice de­part­ments went dark. Governments, which rely on the same ca­bles as every­one else for the vast ma­jor­ity of their com­mu­ni­ca­tions, would be largely cut off from their over­seas out­posts and each other. Satellites would not be able to pick up even half a per­cent of the traf­fic. Contemplating the prospect of a mass ca­ble cut to the UK, then-MP Rishi Sunak con­cluded, “Short of nu­clear or bi­o­log­i­cal war­fare, it is dif­fi­cult to think of a threat that could be more jus­ti­fi­ably de­scribed as ex­is­ten­tial.”

Fortunately, there is enough re­dun­dancy in the world’s ca­bles to make it nearly im­pos­si­ble for a well-con­nected coun­try to be cut off, but ca­ble breaks do hap­pen. On av­er­age, they hap­pen every other day, about 200 times a year. The rea­son web­sites con­tinue to load, bank trans­fers go through, and civ­i­liza­tion per­sists is be­cause of the thou­sand or so peo­ple liv­ing aboard 20-some ships sta­tioned around the world, who race to fix each ca­ble as soon as it breaks.

Photo by Go Takayama for The Verge

Grapnels on the fore­deck of the Ocean Link.

Photo by Go Takayama for The Verge

“Mushroom” an­chors, used in­stead of fluked an­chors to avoid en­tan­gling ca­bles.

Photo by Go Takayama for The Verge

Bow sheave on the Ocean Link, where ca­bles and grap­nel ropes pass over into the sea.

View of the Ocean Link bridge from the fore­deck.

Photo by Go Takayama for The Verge

The in­dus­try re­spon­si­ble for this cru­cial work traces its ori­gins back far be­yond the in­ter­net, past even the tele­phone, to the early days of teleg­ra­phy. It’s in­vis­i­ble, un­der­ap­pre­ci­ated, ana­log. Few peo­ple set out to join the pro­fes­sion, mostly be­cause few peo­ple know it ex­ists.

Hirai’s ca­reer path is char­ac­ter­is­tic in its cir­cuitous­ness. Growing up in the 1960s in the in­dus­trial city of Yokosuka, just down the Miura Peninsula from the Ocean Link’s port in Yokohama, he worked at his par­ents’ fish mar­ket from the age of 12. A teenage love of American rock ‘n’ roll led to a de­sire to learn English, which led him to take a job at 18 as a switch­board op­er­a­tor at the tele­com com­pany KDDI as a means to prac­tice. When he was 26, he trans­ferred to a ca­ble land­ing sta­tion in Okinawa be­cause work­ing on the beach would let him per­fect his wind­surf­ing. This was his in­tro­duc­tion to ca­ble main­te­nance and also where he met his wife. Six years later, his English pro­fi­ciency got him called back to KDDI head­quar­ters to help de­sign Ocean Link for KCS, a KDDI sub­sidiary. Once it was built, he de­cided to go to sea with it, even­tu­ally be­com­ing the ship’s chief en­gi­neer.

Captain Shoichi Suzuki in the bridge of the Ocean Link.

Photo by Go Takayama for The Verge

Others come to the field from mer­chant navies, ma­rine con­struc­tion, ca­ble en­gi­neer­ing, ge­ol­ogy, op­tics, or other tan­gen­tially re­lated dis­ci­plines. When Fumihide Kobayashi, the sub­mersible op­er­a­tor — a tall and solidly built man from the moun­tain re­gion of Nagano — joined KCS at the age of 20, he thought he would be work­ing on ship main­te­nance, not work­ing aboard a main­te­nance ship. He had never been on a boat be­fore, but Hirai en­ticed him to stay with sto­ries of all the whales and other ma­rine crea­tures he would see on the re­mote ocean.

Once peo­ple are in, they tend to stay. For some, it’s the ad­ven­ture — re­pair­ing ca­bles in the churn­ing cur­rents of the Congo Canyon, en­dur­ing hull-dent­ing North Atlantic storms. Others find a sense of pur­pose in main­tain­ing the in­fra­struc­ture on which so­ci­ety de­pends, even if most peo­ple’s re­sponse when they hear about their job is, But is­n’t the in­ter­net all satel­lites by now? The sheer scale of the work can be thrilling, too. People will some­times note that these are the largest con­struc­tion pro­jects hu­man­ity has ever built or sum up a decades-long re­sume by say­ing they’ve laid enough ca­ble to cir­cle the planet six times.

KCS has around 80 em­ploy­ees, many of whom, like Hirai, have worked there for decades. Because the in­dus­try is small and ca­reers long, it can seem like every­one knows one an­other. People of­ten re­fer to it as a fam­ily. Shipboard life lends it­self to a strong sense of ca­ma­raderie, with pe­ri­ods of col­lab­o­ra­tion un­der pres­sure fol­lowed by long stretches — en route to a work­site or wait­ing for storms to pass — with­out much to do but hang out. Kobayashi learned to fish off the side of the ship and at­tempted to im­prove the repet­i­tive cui­sine by serv­ing his crew­mates sashimi. (His fa­vorite is squid, but his col­leagues would pre­fer he use the squid to catch mack­erel.) Hirai, an en­thu­si­as­tic ath­lete, fig­ured out how to string up a net on the Ocean Link’s he­lideck and play ten­nis. Other times, he would join the crew for karaoke in the lounge, a wood-pan­eled room be­hind an anom­alous stained-glass door con­tain­ing mas­sage chairs, a DVD li­brary, and a bar. A self-de­scribed “walking juke­box,” Hirai fa­vored Simon & Garfunkel and Billy Joel, though he said the younger mem­bers of the fleet did­n’t go in for it as much.

Photo by Go Takayama for The Verge

Photo by Go Takayama for The Verge

Photo by Go Takayama for The Verge

Photo by Go Takayama for The Verge

The world is in the midst of a ca­ble boom, with mul­ti­ple new transoceanic lines an­nounced every year. But there is grow­ing con­cern that the in­dus­try re­spon­si­ble for main­tain­ing these ca­bles is run­ning per­ilously lean. There are 77 ca­ble ships in the world, ac­cord­ing to data sup­plied by SubTel Forum, but most are fo­cused on the more prof­itable work of lay­ing new sys­tems. Only 22 are des­ig­nated for re­pair, and it’s an ag­ing and eclec­tic fleet. Often, main­te­nance is their sec­ond act. Some, like Alcatel’s Ile de Molene, are con­verted tugs. Others, like Global Marine’s Wave Sentinel, were once fer­ries. Global Marine re­cently told Data Centre Dynamics that it’s try­ing to ex­tend the life of its ships to 40 years, cit­ing a lack of money. One out of 4 re­pair ships have al­ready passed that mile­stone. The de­sign life for bulk car­ri­ers and oil tankers, by con­trast, is 20 years.

“We’re all happy to spend bil­lions to build new ca­bles, but we’re not re­ally think­ing about how we’re go­ing to look af­ter them,” said Mike Constable, the for­mer CEO of Huawei Marine Networks, who gave a pre­sen­ta­tion on the state of the main­te­nance fleet at an in­dus­try event in Singapore last year. “If you talk to the ship op­er­a­tors, they say it’s not sus­tain­able any­more.”

He pointed to a case last year when four of Vietnam’s five sub­sea ca­bles went down, slow­ing the in­ter­net to a crawl. The ca­bles had­n’t fallen vic­tim to some cat­a­strophic event. It was just the usual en­tropy of fish­ing, ship­ping, and tech­ni­cal fail­ure. But with nearby ships al­ready busy on other re­pairs, the ca­bles did­n’t get fixed for six months. (One promptly broke again.)

But per­haps a greater threat to the in­dus­try’s long-term sur­vival is that the peo­ple, like the ships, are get­ting old. In a pro­fes­sion learned al­most en­tirely on the job, peo­ple take longer to train than ships to build.

A pow­er­ful but del­i­cate 12-foot di­am­e­ter elec­tro-hy­draulic steel drum used for pay­ing out and re­cov­er­ing ca­bles and grap­nels dur­ing re­pairs.

A con­veyor com­prised of 21 pairs of ca­ble-grip­ping tires used for lay­ing and re­triev­ing ca­bles.

A com­mand cen­ter ad­join­ing the bridge where ca­ble ten­sion is mon­i­tored and all ca­ble op­er­a­tions are man­aged.

Three tanks ca­pa­ble of hold­ing a to­tal of 2,800 miles of ca­ble.

A rolling sheave that ca­bles and grap­nel ropes are passed over.

Bow and stern thrusters are used to ma­neu­ver into wind, waves, and cur­rents to keep the ship sta­tion­ary dur­ing re­pairs.

Remote sub­mersible ca­pa­ble of op­er­at­ing at up to 8,000ft. Equipped with cam­eras, sen­sors, a ro­botic arm, and a pow­er­ful wa­ter jet for bury­ing ca­bles.

A pow­er­ful but del­i­cate 12-foot di­am­e­ter elec­tro-hy­draulic steel drum used for pay­ing out and re­cov­er­ing ca­bles and grap­nels dur­ing re­pairs.

A con­veyor com­prised of 21 pairs of ca­ble-grip­ping tires used for lay­ing and re­triev­ing ca­bles.

A com­mand cen­ter ad­join­ing the bridge where ca­ble ten­sion is mon­i­tored and all ca­ble op­er­a­tions are man­aged.

Three tanks ca­pa­ble of hold­ing a to­tal of 2,800 miles of ca­ble.

A rolling sheave that ca­bles and grap­nel ropes are passed over.

Bow and stern thrusters are used to ma­neu­ver into wind, waves, and cur­rents to keep the ship sta­tion­ary dur­ing re­pairs.

Remote sub­mersible ca­pa­ble of op­er­at­ing at up to 8,000ft. Equipped with cam­eras, sen­sors, a ro­botic arm, and a pow­er­ful wa­ter jet for bury­ing ca­bles.

“One of the biggest prob­lems we have in this in­dus­try is at­tract­ing new peo­ple to it,” said Constable. He re­called an­other panel he was on in Singapore meant to in­tro­duce uni­ver­sity stu­dents to the in­dus­try. “The au­di­ence was prob­a­bly about 10 uni­ver­sity kids and 60 old gray peo­ple from the in­dus­try just fill­ing out their day,” he said. When he speaks with stu­dents look­ing to get into tech, he tries to con­vince them that sub­sea ca­bles are also part — a foun­da­tional part — of the tech in­dus­try. “They all want to be data sci­en­tists and that sort of stuff,” he said. “But for me, I find this in­dus­try fas­ci­nat­ing. You’re deal­ing with the most hos­tile en­vi­ron­ment on the planet, eight kilo­me­ters deep in the oceans, work­ing with some pretty high tech­nol­ogy, trav­el­ing all over the world. You’re on the fore­front of geopol­i­tics, and it’s crit­i­cal for the whole way the world op­er­ates now.”

The lifestyle can be an ob­sta­cle. A ca­reer in sub­sea means en­dur­ing long stretches far from home, un­pre­dictable sched­ules, and iron­i­cally, very poor in­ter­net.

Photo by Go Takayama for The Verge

“Everyone com­plains about that,” said Kaida Takashi, a se­nior ad­vi­sor at KCS, who is try­ing to get the Ocean Link set up with Starlink. It’s a gen­er­a­tional dif­fer­ence, he said. For some­one like him, a 62-year-old ham ra­dio en­thu­si­ast, Wi-Fi barely fast enough to email is a lux­ury. Other in­dus­try vet­er­ans rem­i­nisced about the days when they felt for­tu­nate to get faxes on board, or wait­ing for the mail­bag in port, or the nov­elty of us­ing the very ca­ble they were lay­ing to make calls from the mid­dle of the ocean. But for peo­ple who grew up with an ex­pec­ta­tion of con­stant con­nec­tiv­ity, the dis­con­nec­tion of ship­board life can cause vis­i­ble dis­com­fort. “It’s a part of them,” one in­dus­try vet­eran mar­veled of his younger col­leagues. “They can’t let it go.”

The in­dus­try’s biggest re­cruit­ing chal­lenge, how­ever, is the in­dus­try’s in­vis­i­bil­ity. It’s a tru­ism that peo­ple don’t think about in­fra­struc­ture un­til it breaks, but they tend not to think about the fix­ing of it, ei­ther. In his 2014 es­say, “Rethinking Repair,” pro­fes­sor of in­for­ma­tion sci­ence Steven Jackson ar­gued that con­tem­po­rary think­ing about tech­nol­ogy ro­man­ti­cizes mo­ments of in­ven­tion over the on­go­ing work of main­te­nance, though it is equally im­por­tant to the de­ploy­ment of func­tional tech­nol­ogy in the world. There are few bet­ter ex­am­ples than the sub­sea ca­ble in­dus­try, which, for over a cen­tury, has been so ef­fec­tive at quickly fix­ing faults that the pub­lic has rarely had a chance to no­tice. Or as one in­dus­try vet­eran put it, “We are one of the best-kept se­crets in the world, be­cause things just work.”

TheThe Ocean Link spent two nights at sea be­fore re­ceiv­ing or­ders to re­turn. As they neared land, Hirai saw de­bris from the tsunami’s back­wash float­ing in the wa­ter: fish­ing nets, tires, the roofs of build­ings, the bloated body of what he guessed was a cow.

The earth­quake mea­sured 9.1 on the Richter scale, the fourth largest ever recorded and the largest to ever hit Japan. But it was the se­ries of tsunami waves that ar­rived half an hour later that dealt the most de­struc­tion, surg­ing miles in­land and sweep­ing build­ings, cars, and thou­sands of peo­ple out to sea. The death toll would even­tu­ally climb to nearly 20,000, and the day would be­come a na­tional tragedy re­ferred to sim­ply as “3/11.”

The full ex­tent of the dev­as­ta­tion was still be­com­ing clear when the Ocean Link re­turned, but the dis­as­ter had al­ready en­tered a new phase. One hun­dred and sixty miles north of Tokyo, a 50-foot tsunami wave over­topped a sea­wall pro­tect­ing the Fukushima power plant, swamp­ing the emer­gency gen­er­a­tors that were cool­ing the re­ac­tors through its au­to­matic post-quake shut­down and pre­cip­i­tat­ing a nu­clear melt­down.

Hirai’s wife and son had made it back home to their house in Yokosuka, where they lived with Hirai’s par­ents. Kobayashi’s fam­ily, too, was safe. Some crew lost loved ones; oth­ers sent fam­ily to stay with rel­a­tives in the south out of fear of ra­di­a­tion. They all knew that they had only a few days be­fore they would be sent back out to sea.

The Ocean Link in a storm in the North Pacific. Sometimes, Hirai said, storms are so bad you can’t work or sleep. All you can do is hold onto your bunk and laugh.

The Ocean Link in a storm in the North Pacific. The ship pitches wildly in the heavy swell, the waves crash­ing over its bow.

The dis­as­ter had sev­ered phone lines and wrecked cell tow­ers, caus­ing phone ser­vice to cut out al­most im­me­di­ately af­ter the earth­quake struck. Instead, peo­ple turned to email, Skype, and other on­line ser­vices that were mostly able to route around dam­age to the net­work. There was a sense, ac­cord­ing to one en­gi­neer’s post­mortem pre­sen­ta­tion, that the in­ter­net was the only me­dia that sur­vived.

But its sur­vival was more ten­u­ous than the pub­lic knew. While the ca­bles con­nect­ing Japan to the rest of the world sur­vived the ini­tial de­struc­tion, later that night, as mil­lions of peo­ple tried to find their way home with trains stopped and power in­ter­mit­tent, en­gi­neers in Tokyo net­work op­er­a­tion cen­ters watched as one ca­ble af­ter an­other failed. By the next morn­ing, seven of Japan’s 12 transpa­cific ca­bles were sev­ered. Engineers work­ing through the night and fol­low­ing days man­aged to shift traf­fic to those that re­mained, but the new routes were near their max­i­mum ca­pac­ity. The head of tele­com com­pany NTT’s op­er­a­tion cen­ter at the time es­ti­mated that if an­other ca­ble failed, it would have lost all traf­fic to the US. With servers for most ma­jor in­ter­net com­pa­nies lo­cated there, Japan would have ef­fec­tively lost the in­ter­net.

Normally, the se­quence of re­pairs would be de­ter­mined by whichever ca­ble owner re­ported the fault first, but given the ex­tra­or­di­nary cir­cum­stances, the usu­ally self-in­ter­ested ca­ble own­ers agreed to de­fer to KCS. The pri­or­ity was to re­pair a ca­ble — any ca­ble — as fast as pos­si­ble.

It was im­pos­si­ble to know the state of the ca­bles on the ocean floor, so like foren­sic in­ves­ti­ga­tors, Hirai and the other en­gi­neers had to work with the sparse facts avail­able. By hav­ing the ca­ble land­ing sta­tions on ei­ther side of the ocean beam light down their end of the line and time the re­flec­tions back, they were able to lo­cate the faults near­est to them within a few me­ters. Most of the faults lay in deep wa­ter, in the canyons chan­nel­ing into the Japan Trench. This, plus the tim­ing of the faults, in­di­cated it was­n’t the quake that broke them but the un­der­wa­ter avalanches it trig­gered.

“It has­n’t changed in 150 years… The Victorians did it that way and we’re do­ing it the same way.”

Submarine land­slides are awe­some events whose ex­is­tence was only dis­cov­ered in the 1950s, when sci­en­tists an­a­lyzed the tim­ing of 12 ca­ble faults that sev­ered com­mu­ni­ca­tion be­tween Europe and North America two decades ear­lier. Before then, ac­cord­ing to oceanog­ra­pher Mike Clare, “It was as­sumed that deep wa­ter was bor­ing and noth­ing hap­pens down there.” In fact, the ocean floor is riven with moun­tains and canyons that ex­pe­ri­ence avalanches that dwarf any­thing found on land, cas­cades of sed­i­ment and de­bris rac­ing for hun­dreds of miles. Hirai had dealt with them in Taiwan in 2006, one of the most no­to­ri­ous events in the an­nals of ca­ble re­pair.

On December 26th, an earth­quake dis­lodged sed­i­ment on Taiwan’s south­ern coast and sent it rush­ing 160 miles into the Luzon Strait, one of sev­eral global ca­ble choke­points. Nine ca­bles were sev­ered and Taiwan was knocked al­most en­tirely of­fline. Banking, air­lines, and com­mu­ni­ca­tions were dis­rupted through­out the re­gion. Trading of the Korean won was halted. The ca­bles, buried un­der moun­tains of de­bris, were nearly im­pos­si­ble to find. It took 11 ships, in­clud­ing the Ocean Link, nearly two months to fin­ish re­pairs.

Often in a multi-ca­ble dis­as­ter like the Taiwan earth­quake, every ship in the re­gion comes to as­sist. But with Japan, there was an un­prece­dented com­pli­ca­tion: the ma­jor­ity of the faults were lo­cated off­shore of the on­go­ing nu­clear melt­down at Fukushima. Ship op­er­a­tors deemed as­sis­tance too risky, which meant that, for the time be­ing, the Ocean Link was on its own.

The crew felt not only duty bound to work but uniquely ca­pa­ble of do­ing so. They had dealt with ra­di­a­tion be­fore, though not at this scale. In 1993, shortly be­fore the Ocean Link was to lay a ca­ble link­ing Japan, Korea, and Russia, they learned the Soviets had dumped ra­dioac­tive waste in the ocean along the planned route. With some trep­i­da­tion, KCS pro­ceeded with the job. They bought Geiger coun­ters and pro­tec­tive gear, flew in nurses from the US with chem­i­cal weapons train­ing, and scanned the wa­ter for ra­di­a­tion as they went. When none was de­tected, they put the gear in stor­age.

Now, as they read­ied the ship for de­par­ture, an em­ployee was dis­patched to the de­pot to find the old ra­di­a­tion gear. A lo­cal uni­ver­sity do­nated a few more sen­sors and trained the crew on how to use them.

They de­cided to be­gin with the same ca­ble they had just fin­ished re­pair­ing when the earth­quake struck. On a driz­zling af­ter­noon eight days af­ter re­turn­ing to port, with smoke still ris­ing from the Fukushima power plant, the Ocean Link set back out to sea.

Photo by Go Takayama for The Verge

ToTo the ex­tent he is re­mem­bered, Cyrus Field is known to his­tory as the per­son re­spon­si­ble for run­ning a tele­graph ca­ble across the Atlantic Ocean, but he also con­ducted what at the time was con­sid­ered an equally great tech­ni­cal feat: the first deep-sea ca­ble re­pair.

Field, a 35-year-old self-made pa­per ty­coon, had no ex­pe­ri­ence in teleg­ra­phy — which helps ex­plain why, in 1854, he em­barked on such a quixotic mis­sion. Though small bod­ies of wa­ter like the English Channel had been bridged by tele­graph, fail­ure was rou­tine and costly. Cables shorted out, snapped un­der ten­sion, snagged on rocks, were sliced by an­chors, twisted by cur­rents, tan­gled around whales, at­tacked by sword­fish, and de­voured by a “miserable lit­tle mol­lusc” called the Teredo worm with an ap­petite for jute in­su­la­tion.

Field fared no bet­ter. Twelve years af­ter he be­gan, he had en­dured sev­ered ca­bles, near sink­ings, and had one “success”: a ca­ble laid in 1858 that prompted cel­e­bra­tions so en­thu­si­as­tic that rev­el­ers set fire to New York City Hall. The ca­ble failed weeks later.

The SS Great Eastern at­tempt­ing to re­cover the bro­ken transat­lantic tele­graph ca­ble in 1865.

Sailors coil­ing the transat­lantic tele­graph ca­ble aboard the SS Great Eastern in 1865.

Pieces of the first transat­lantic tele­graph ca­bles and a model of the grap­nel used to re­cover them.

Cyrus West Field, who fi­nanced and or­ga­nized the lay­ing of the first transat­lantic tele­graph ca­bles.

Field tried again seven years later only for the ca­ble to snap halfway across the Atlantic. The next year, he set out yet again, promis­ing not only to fi­nally lay a work­ing transat­lantic ca­ble but to re­cover the bro­ken ca­ble and fin­ish that one, too.

By that time, a crude method had been de­vel­oped for fix­ing ca­bles in shal­low wa­ter. A ship would drag a hooked grap­nel an­chor across the seafloor, un­til, like the tremor of a fish­ing line, in­creas­ing ten­sion showed they’d caught the ca­ble, which they would then haul on board to fix. Field’s plan was ba­si­cally this but big­ger: big­ger hooks, stronger rope, more pow­er­ful wind­ing en­gine, all aboard the largest ship afloat, a pas­sen­ger liner called the SS Great Eastern that had been retro­fit­ted for the mis­sion. William Thomson, the pro­jec­t’s sci­en­tific ad­viser and the fu­ture Lord Kelvin, did the math and deemed it fea­si­ble.

The fol­low­ing ex­change can be found in the sec­ond vol­ume of Letters of Note. Reprinted by kind per­mis­sion of the Reagan Library. The pic­ture of Ronald Reagan peer­ing into a child’s messy bed­room—which I’m now re­al­is­ing, as I type, is mildly sin­is­ter?—is in fact, be­lieve it or not, two pho­tos mashed to­gether. Both are from Getty who I’m sure won’t mind.

As one would ex­pect, Ronald Reagan was the re­cip­i­ent of thou­sands of let­ters each month dur­ing his pres­i­dency; a mail­bag so vo­lu­mi­nous, in fact, that a gang of pa­tient vol­un­teers were tasked with open­ing them all on his be­half and pass­ing him ap­prox­i­mately 30 each week to read and re­spond to. Letters ar­rived from all over the world, writ­ten by a di­verse group of peo­ple: men, women, fans, crit­ics, av­er­age Joes, celebri­ties, world lead­ers, and, mark­ing a mo­ment in his­tory, a let­ter from a 13-year-old boy from South Carolina named Andy Smith, writ­ten ex­actly 40 years ago on 18 April 1984.

My name is Andy Smith. I am a sev­enth grade stu­dent at Irmo Middle School, in Irmo, South Carolina. Today my mother de­clared my bed­room a dis­as­ter area. I would like to re­quest fed­eral funds to hire a crew to clean up my room. I am pre­pared to pro­vide the ini­tial funds if you will pro­vide match­ing funds for this pro­ject.I know you will be fair when you con­sider my re­quest. I will be await­ing your re­ply.

I’m sorry to be so late in an­swer­ing your let­ter but, as you know, I’ve been in China and found your let­ter here upon my re­turn. Your ap­pli­ca­tion for dis­as­ter re­lief has been duly noted but I must point out one tech­ni­cal prob­lem: the au­thor­ity de­clar­ing the dis­as­ter is sup­posed to make the re­quest. In this case, your mother.How­ever, set­ting that aside, I’ll have to point out the larger prob­lem of avail­able funds. This has been a year of dis­as­ters: 539 hur­ri­canes as of May 4th and sev­eral more since, nu­mer­ous floods, for­est fires, drought in Texas and a num­ber of earth­quakes. What I’m get­ting at is that funds are dan­ger­ously low.May I make a sug­ges­tion? This Administration, be­liev­ing that gov­ern­ment has done many things that could bet­ter be done by vol­un­teers at the lo­cal level, has spon­sored a Private Sector Initiative Program, call­ing upon peo­ple to prac­tice vol­un­tarism in the solv­ing of a num­ber of lo­cal prob­lems.Your sit­u­a­tion ap­pears to be a nat­ural. I’m sure your mother was fully jus­ti­fied in pro­claim­ing your room a dis­as­ter. Therefore, you are in an ex­cel­lent po­si­tion to launch an­other vol­un­teer pro­gram to go along with the more than 3000 al­ready un­der­way in our na­tion. Congratulations.Give my best re­gards to your mother.

In my last post, I dis­cussed some work I had done build­ing Nero, the as­sis­tant of the fu­ture that I’ve al­ways wanted. I ended up cre­at­ing an end-to-end ex­am­ple which used Nx, OpenAI APIs, and ElevenLabs to cre­ate an in-browser home au­toma­tion as­sis­tant. For a first prod­uct, it’s de­cent. Nero is a neat lit­tle party trick that I can use to im­press my non-tech friends. I am, how­ever, not in this busi­ness to im­press my friends. I want Nero to ac­tu­ally help me and ac­tu­ally feel like an as­sis­tant. My pre­vi­ous ver­sion is not that.

One miss­ing piece is the abil­ity to con­verse nat­u­rally with­out browser in­ter­ac­tion. The first im­ple­men­ta­tion of Nero’s “conversational” abil­i­ties re­lied on user in­ter­ac­tion with the screen every time we wanted to ini­ti­ate a re­sponse or ac­tion. Nero also did not re­tain any con­ver­sa­tional his­tory. In short, Nero was not a great con­ver­sa­tional as­sis­tant. It was one of the things I wanted to fix; how­ever, I was mo­ti­vated to do it sooner rather than later af­ter watch­ing an im­pres­sive demo from Retell.

The Retell demo im­ple­ments a con­ver­sa­tional agent backed by their WebSocket API in a browser. The demon­stra­tion has:

* Impressive fil­ter­ing (e.g. snap­ping and other non-voice ac­tiv­ity does­n’t seem to throw off the agent)

Their doc­u­men­ta­tion sug­gests they also have sup­port for backchan­nel­ing and in­tel­li­gent end of turn de­tec­tion—two things that are es­sen­tial to nat­ural con­ver­sa­tional feel but which are very dif­fi­cult to ex­press pro­gram­mat­i­cally.

I had pre­vi­ously con­vinced my­self that I could im­ple­ment a pass­able con­ver­sa­tional agent ex­pe­ri­ence in a short amount of time. So that is what I set out to do.

The first thing that needed to change about Nero’s de­sign was the speech to text pipeline. My orig­i­nal demon­stra­tion re­lied on an ex­am­ple from Bumblebee which im­ple­mented a speech to text pipeline us­ing Whisper. The pipeline uses mouse events in a Phoenix LiveView Hook to start and stop record­ings be­fore send­ing them to the server to ini­ti­ate tran­scrip­tion. If you’re not fa­mil­iar, Phoenix LiveView is a server-side ren­der­ing frame­work built on top of Elixir. LiveView has sup­port for client-side JavaScript hooks which sup­port bidi­rec­tional com­mu­ni­ca­tion be­tween client and server.

The orig­i­nal speech to text im­ple­men­ta­tion used a hook with an event lis­tener at­tached to mouse­down and mouseup on a but­ton to start and stop record­ing. After record­ing stops, the hook de­codes the recorded buffer into a PCM buffer, con­verts the en­di­an­ness, and then pushes the buffer to the server with an up­load. The orig­i­nal hook im­ple­ments most of the func­tion­al­ity we want; how­ever, we need to make some mi­nor tweaks. Rather than trig­ger record­ings to stop and start on mouse events, we want to trig­ger record­ings to start and stop ex­actly when a per­son starts and stops speak­ing. Simple, right?

My first idea in im­ple­ment­ing what I called “always on record­ing” was to mon­i­tor the mi­cro­phone’s vol­ume, and trig­ger a record­ing when the vol­ume reached a cer­tain thresh­old. The record­ing would stop when the vol­ume dipped be­low that thresh­old. At this point, I learned about ge­tUser­Me­dia. ge­tUser­Me­dia prompts the user for per­mis­sion to ac­cess me­dia de­vices such as a mi­cro­phone and/​or we­b­cam, and then pro­duces a MediaStream. A MediaStream is a stream of me­dia con­tent con­tain­ing in­for­ma­tion about au­dio and video tracks in the stream. We can use data from the MediaStream to de­ter­mine speaker ac­tiv­ity and thus trig­ger record­ings.

To de­ter­mine the vol­ume for a given sam­ple, we can use an AnalyserNode. Per the doc­u­men­ta­tion AnyalyserNode is de­signed for pro­cess­ing gen­er­ated au­dio data for vi­su­al­iza­tion pur­poses, but we can use it to de­ter­mine spikes in au­dio:

This uses an analyser and re­peat­edly checks if the vol­ume of the mi­cro­phone at a given frame ex­ceeds the given VOLUME_THRESHOLD. If it does, it checks to see if we are record­ing and if not, starts the record­ing.

After test­ing a bit, I re­al­ized this im­ple­men­ta­tion sucked. Of the many is­sues with this ap­proach, the biggest is that there are many nat­ural dips in a per­son’s vol­ume. Checking a sin­gle frame does­n’t ac­count for these nat­ural dips. To fix this, I thought it would be a good idea to in­tro­duce a time­out which only stopped record­ing af­ter the vol­ume was be­low a thresh­old for a cer­tain amount of time:

This ac­tu­ally ended up work­ing de­cent, but re­quired tun­ing hy­per­pa­ra­me­ters for both VOLUME_THRESHOLD and SILENCE_TIMEOUT. The chal­lenge here is that higher SILENCE_TIMEOUT in­tro­duces ad­di­tion­ally la­tency in tran­si­tion time be­tween a speaker and Nero; how­ever, lower time­outs might be too sen­si­tive to speak­ers with slower and qui­eter speak­ing rhythms. Additionally, a sta­tic VOLUME_THRESHOLD does not ac­count for am­bi­ent noise. Now, de­spite these short­com­ings, I found I was able to pass­ably de­tect a sin­gle speaker in a quiet room.

After hook­ing this up to my ex­ist­ing LiveView and try­ing some end-to-end con­ver­sa­tions, I re­al­ized some­thing was sig­nif­i­cantly off. The tran­scrip­tions I was get­ting were off. I soon re­al­ized that they were al­ways off at the be­gin­ning of a tran­scrip­tion. Shorter au­dio se­quences were es­pe­cially af­fected. It turns out that the de­tec­tion al­go­rithm al­ways re­sulted in some amount of trun­ca­tion at the be­gin­ning of an au­dio clip. When a speaker starts talk­ing, their vol­ume ramps up — it’s not an in­stan­ta­neous spike. To ac­count for this, I in­tro­duced a pre-record­ing buffer which al­ways tracked the pre­vi­ous 150ms of au­dio. After record­ing started, I would stop the pre-record­ing buffer and start the ac­tual record­ing, and then even­tu­ally splice these 2 to­gether to send to the server for tran­scrip­tion.

Overall, this ac­tu­ally worked okay. While there are some ob­vi­ous fail­ure modes, it worked well enough to get a pass­able demon­stra­tion. If you can’t tell by now, I am not an au­dio en­gi­neer. I learned later that this is a very naive at­tempt at voice ac­tiv­ity de­tec­tion. Later on in this post, I’ll run through some of the im­prove­ments I made based on my re­search into the field of VAD.

The demon­stra­tion I built for Nero in my first post al­ready con­tained the scaf­fold­ing for an end-to-end tran­scrip­tion -> re­sponse -> speech pipeline. I only needed to make some slight mod­i­fi­ca­tions to get the phone call demo to work. The end-to-end the pipeline looks like this:

When our al­go­rithm de­tects that speech has stopped, it in­vokes the sto­pRe­cord­ing method. sto­pRe­cord­ing takes the recorded au­dio, does some client-side pre-pro­cess­ing, and up­loads it to the server. The server con­sumes the up­loaded en­try as a part of LiveView’s nor­mal up­loads life­cy­cle and then in­vokes an async task to start tran­scrip­tion:

Note that be­cause we did most of the pre-pro­cess­ing client-side, we can just con­sume the au­dio bi­nary as an Nx. Tensor, with­out any ad­di­tional work. The SpeechToText mod­ule im­ple­ments tran­scrip­tion us­ing Nx.Serving:

Nx. Serving is an ab­strac­tion in the Elixir Nx ecosys­tem for serv­ing ma­chine learn­ing mod­els di­rectly in an Elixir ap­pli­ca­tion. It im­ple­ments dy­namic batch­ing, en­cap­su­lates pre-pro­cess­ing, in­fer­ence, and post-pro­cess­ing, sup­ports dis­tri­b­u­tion and load-bal­anc­ing be­tween mul­ti­ple GPUs na­tively, and in gen­eral is an ex­tremely easy way to serve ma­chine learn­ing mod­els.

After tran­scrip­tion com­pletes, we get an async re­sult we can han­dle to ini­ti­ate a re­sponse:

Here Nero. Agent.respond/1 re­turns an Elixir Stream of text. For my orig­i­nal demon­stra­tion I just used the Elixir OpenAI li­brary to pro­duce a stream from a GPT-3.5 re­sponse:

The re­sponse stream is con­sumed by speak/​2. speak/​2 im­ple­ments the text to speech pipeline:

Where Nero. TextToSpeech.stream/1 uses the ElevenLabs WebSocket API to stream text in and speech out. You can read a bit more about the im­ple­men­ta­tion in my pre­vi­ous post.

Nero. TextToSpeech.stream/1 re­turns the con­sumed re­sponse as text so we can ap­pend that to the chat his­tory af­ter the :speak task fin­ishes:

This is ba­si­cally all of the scaf­fold­ing needed for an end-to-end demo, but I wanted to add a few more fea­tures. First, I wanted to sup­port “intelligent” hang-ups. Basically, I wanted to be able to de­tect when a con­ver­sa­tion was fin­ished, and stop the record­ing. To do that, I used Instructor:

Please ig­nore my won­der­fully en­gi­neered prompt. This uses GPT-3.5 to de­ter­mine whether or not a given con­ver­sa­tion has ended. After every one of Nero’s turns, we check the tran­script to pos­si­bly end the call:

This pushes a hang_up event to the socket:

Which stops the record­ing, and then pushes an event to tog­gle_­con­ver­sa­tion back to the server. tog­gle_­con­ver­sa­tion im­ple­ments the start/​stop logic from the server:

Finally, I wanted to im­ple­ment in­for­ma­tion ex­trac­tion from the tran­script. Again, I used in­struc­tor and de­fined an ex­trac­tion schema:

And used GPT-3.5 with a rough prompt to get the nec­es­sary in­for­ma­tion from the tran­script:

And then any­time a con­ver­sa­tion ends, we at­tempt to re­trieve ap­point­ment in­for­ma­tion:

Now this is es­sen­tially the ex­act im­ple­men­ta­tion that pro­duced this demon­stra­tion. End-to-end this amounted to a cou­ple of hours of work; how­ever, I al­ready had most of the ba­sic scaf­fold im­ple­mented from my pre­vi­ous work on Nero. In my bi­ased opin­ion, I think my demo is pretty good, but as oth­ers have pointed out Retell’s demo kicks my ass in:

And so, I set out to im­prove my im­ple­men­ta­tion — start­ing with la­tency.

Human con­ver­sa­tions have ex­tremely tight “time-to-turn.” In-person con­ver­sa­tions are es­pe­cially rapid be­cause we rely on vi­sual as well as au­dio sig­nals to de­ter­mine when it’s our time to par­tic­i­pate in a con­ver­sa­tion. The “average” time-to-turn in a con­ver­sa­tion can be as quick as 200ms. That means for a con­ver­sa­tional agent to feel re­al­is­tic, it needs an ex­tremely quick turn around time for “time to first spo­ken word.”

After post­ing my orig­i­nal demon­stra­tion, I al­ready knew there were some very easy op­ti­miza­tions I could make, so I set out to im­prove the av­er­age la­tency of my im­ple­men­ta­tion as much as pos­si­ble in a short amount of time. First, I needed at least some method for de­ter­min­ing whether an op­ti­miza­tion worked. My rudi­men­tary ap­proach was to use JavaScript Performance Timers and log­ging. Basically, I com­puted a start­Time from the ex­act mo­ment an au­dio record­ing stopped and an end­Time from the ex­act mo­ment an au­dio out­put started, and then I logged that time to the con­sole.

This is a very un­sci­en­tific way of do­ing busi­ness. In the fu­ture, I’d like to im­ple­ment a much more in­volved pro­fil­ing and bench­mark­ing method­ol­ogy. For this process though, it worked well enough.

Next, I con­sid­ered all of the ar­eas that could in­tro­duce la­tency into the pipeline. From the mo­ment a record­ing stops, these are all of the steps we take:

Pre-process record­ing by con­vert­ing to PCM buffer, and then con­vert­ing en­di­an­ness to match server (if nec­es­sary)

Perform speech to text on buffer to pro­duce text

That’s a lot of steps that can in­tro­duce la­tency, in­clud­ing po­ten­tially 3 (in our case 2 be­cause we own the STT pipeline) net­work calls.

Next, I wanted to esab­lish a “baseline” of per­for­mance. To demon­strate this it­er­a­tive process, I did a base­line ex­am­ple on my M3 Mac CPU. Note that this is go­ing to be slow rel­a­tive to my pre­vi­ous demo be­cause the pre­vi­ous demo runs on a GPU. The base­line per­for­mance I got from the orig­i­nal demo run­ning on my mac was 4537 ms. 4.5 sec­onds turn around time. Yikes. Lots of work to do.

To start, I knew that the SILENCE_TIMEOUT used to wait for speech to end was rather long. For the orig­i­nal demo, I used 1000 ms, which ba­si­cally means a speaker has to stop talk­ing for a full sec­ond be­fore we’ll even start the long re­sponse process. After some trial and er­ror, I fig­ured 500 ms was a “passable” hy­per­pa­ra­me­ter. After ad­just­ing this down, the la­tency change was al­most ex­actly cor­re­lated to the dip: 4079 ms.

I had a hunch that my text to speech pipeline was not ef­fi­cient. Fortunately, ElevenLabs gives us a nice Latency Guide. The first sug­ges­tion is to use their turbo model by spec­i­fy­ing eleven_­tur­bo_v2. I set that and we got a slight per­for­mance boost: 4014 ms.

Next, they sug­gest adding op­ti­mize_stream­ing_la­tency. I set the value to 3 and we get: 3791 ms. Their next sug­ges­tion is to use a pre-made voice. I ac­tu­ally did­n’t re­al­ize un­til much later that I was not us­ing a pre-made voice so I don’t have a com­par­i­son for how that change im­pacted la­tency.

Now it says to limit clos­ing WebSocket con­nec­tions. my cur­rent im­ple­men­ta­tion opens a con­nec­tion every­time it speaks — which is not good. Basically every “turn” has to es­tab­lish a new web­socket con­nec­tion. Additionally, ElevenLabs has a time­out of 20s from when you con­nect. So you need to send a mes­sage at least every 20s. I con­sid­ered 2 op­tions at this point:

Open a global WebSocket con­nec­tion, or maybe even a pool of con­nec­tions, and try to keep the con­nec­tion alive. But that seems re­ally waste­ful, and I don’t think is the in­tended use of their API

Open a WebSocket con­nec­tion when convo starts. We don’t have to worry about 20s pauses

I de­cided to go with op­tion 2, but I still think there are some draw­backs and con­sid­er­a­tions for a pro­duc­tion sys­tem. The im­ple­men­ta­tion I used opens a web­socket con­nec­tion on first “speak” and stores the con­nec­tion PID as an as­sign in the LiveView socket. If you have a sys­tem with po­ten­tially many con­cur­rent users speak­ing, you run the risk of cre­at­ing a po­ten­tially un­bounded num­ber of con­nec­tions. A more ro­bust so­lu­tion would prob­a­bly use con­nec­tion pools; how­ever, I’m not re­ally wor­ried about traf­fic or scal­ing here.

While adding this op­ti­miza­tion, I strug­gled a bit be­cause ElevenLabs would send the first frame back, then cut off. Then I re­al­ized that it was wait­ing to gen­er­ate be­cuase it thought I was go­ing to send more frames. So I needed to “flush” the gen­er­a­tion af­ter I fin­ished send­ing my to­kens. This also seemed to fix un­nat­ural au­dio prob­lems I was hav­ing. After ap­ply­ing this op­ti­miza­tion, our time to first spo­ken word was slightly lower in the 3700 ms range.

After pe­rus­ing their docs a bit more, I learned that ElevenLabs will send PCM buffers in­stead of MP3. Web Browser’s have to de­code MP3 to PCM, which po­ten­tially in­tro­duces some over­head. One draw­back is that you need to be on the in­de­pen­dent cre­ator tier to re­ceive PCM in­stead of MP3. Now, if you’re won­der­ing if I spent $99 to save some mil­lisec­onds for a mean­ing­less demo, the an­swer is ab­solutely yes I did.

At this point, I be­lieve I’ve ex­hausted a lot of the “easy” op­ti­miza­tions for TTS la­tency. One thing that does bother me about the ElevenLabs Websocket API is that there’s no way to re­ceive bi­nary pay­loads in­stead of JSON pay­loads. This is prob­a­bly be­cause they send align­ment data, but I’m not us­ing the align­ment data here. When han­dling an in­com­ing frame from their API we have to first de­code the JSON, and then de­code the Base64 en­coded au­dio buffer. I’m not sure what the la­tency im­pact is, but I’m sure we could shave some time by avoid­ing both of these con­ver­sions. I also think the Base64 rep­re­sen­ta­tion re­sults in slightly larger buffers which could im­pact net­work la­tency.

The next area I looked to im­prove was the speech-to-text pipeline. I am us­ing Nx. Serving specif­i­cally for Speech-to-Text. The ben­e­fit of this ap­proach is that we can avoid an ad­di­tional net­work call just for tran­scrip­tion. Of course, that as­sumes our tran­scrip­tion pipeline can run fast enough on our own hard­ware. XLA is no­to­ri­ously slow on CPUs (it’s get­ting bet­ter). The first “optimization” I did was to switch to my GPU: 2050 ms

And that right there is a bit­ter les­son, be­cause it’s the largest per­for­mance boost we’re go­ing to get.

Next, I re­al­ized the model is­n’t us­ing F16, which can in­tro­duce some solid speed-ups: 1800 ms. Now, there are prob­a­bly some ad­di­tional op­ti­miza­tions we could add to Nx and EXLA specif­i­cally. For ex­am­ple, we don’t have a flash at­ten­tion im­ple­men­ta­tion. Of course, XLA does a great job of ap­ply­ing sim­i­lar op­ti­miza­tions as a base­line, so I’m not sure how much it would help. There’s also fast JAX im­ple­men­ta­tions of Whisper that claim up to 70x speed ups. One is­sue with a lof of these claimed speed-ups; how­ever, is that they are al­most al­ways for long au­dio se­quences. GPUs and TPUs do well with large batch sizes and se­quence lengths, but not for batch size 1 and short se­quence lengths like we care about in this im­ple­men­ta­tion. One day I may go down the per­for­mance hole of fast batch size 1 tran­scrip­tion, but to­day is not that day.

At this point, I had moved on to im­prov­ing some of the fail­ure modes of my demo. While do­ing so, I learned much more about au­dio than I had pre­vi­ously known, and re­al­ized that the con­fig­u­ra­tion I used to record au­dio can sig­nif­i­cantly im­prove whis­per per­for­mance as well. Turns out there’s a nice guide of some­body dis­cussing pa­ra­me­ters that work. Specifically, you should use 16 kHz sam­ple rate for tran­scrip­tions. Reducing the sam­ple rate also should re­duce net­work over­head be­cause we have less data, but it could re­duce qual­ity of the tran­scrip­tion. Oh well. Additionally, I re­al­ized I was­n’t us­ing a pre-made ElevenLabs voice. After in­tro­duc­ing both of these op­ti­miza­tions, I was able to achieve 1520 ms turn time.

Finally, I re­al­ized I was do­ing all of my bench­marks on a de­vel­op­ment server. I switched my phoenix en­vi­ron­ment from dev to prod and got: 1375 ms. So, with all of these op­ti­miza­tions we’re sit­ting at about 1.3s turn around time in a con­ver­sa­tion. When con­vers­ing, it starts to feel some­what close to nat­ural. I should also point out that this is also run­ning over Tailscale, so there is about 100 ms ping be­tween my Mac and the server run­ning on my GPU. When I run this lo­cally on my GPU, I can con­sis­tently get about 1000 ms and some­times 900 ms turn around time. Still, un­for­tu­nately, this does not match Retell’s la­tency. According to them, they are able to achieve 800 ms con­sis­tently. I have some mus­ings at the end about how this is pos­si­ble.

I be­lieve the biggest area I could im­prove the im­ple­men­ta­tion is to use a bet­ter VAD im­ple­men­ta­tion that re­lies on small rolling win­dows of ac­tiv­ity rather than frames. We could prob­a­bly get away with us­ing 20-30 ms win­dows, which could the­o­ret­i­cally of­fer a 480 ms la­tency im­prove­ment. I would like to even­tu­ally ex­plore this.

In all hon­esty though, I think that is a sig­nif­i­cant im­prove­ment, and I could prob­a­bly stop right here and be done with it.

If I were to keep go­ing, I would ex­plore us­ing a lo­cal LLM with Nx and Bumblebee. Nx and Bumblebee sup­port LLMs like Mistral and Llama out-of-the box. And our text gen­er­a­tion serv­ings sup­port stream­ing. That means we can pos­si­bly elim­i­nate any net­work la­tency to OpenAI, and in­stead run 2 of the 3 mod­els lo­cally. One is­sue with this is that Nx cur­rently does not have any quan­tized in­fer­ence sup­port (it’s com­ing I promise), so my sin­gle 4090 is not suf­fi­cient to de­ploy both Whisper and Mistral. Fortunately, the folks at Fly.io were kind enough to give me ac­cess to some 80GB A100s. I will post a demo when I get one de­ployed 🙂

Maybe one day I will im­ple­ment StyleTTS2 and see how ef­fi­cient we can get with an en­tirely lo­cal in­fer­ence pipeline.

Some peo­ple pointed out that my orig­i­nal demo did not have the same con­ver­sa­tional ex­pe­ri­ence as Retell’s, and they are ab­solutely right. Aside from la­tency, mine was prone to fail­ure, picks up sys­tem sounds, picks up ran­dom noises like key­board and mouse clicks, and does­n’t do well with am­bi­ent noise. They also have sup­port for backchan­nel­ing, fillers and in­ter­rup­tions which in­tro­duces some el­e­ment of “realness” when in­ter­act­ing with their agent.

Now I did­n’t get around to adding backchan­nels or fillers, but I was able to make some slight im­prove­ments to the VAD al­go­rithm I used, and I added sup­port for in­ter­rup­tions.

The first fail­ure mode that seems to hap­pen is echo from the sys­tem sounds. Nero al­ways records and will start tran­scrib­ing af­ter au­dio spikes over a cer­tain thresh­old. After some dig­ging into the ge­tUser­Me­dia API, I found op­tions for echoCan­cel­la­tion, nois­eSup­pres­sion, and au­to­Gain­Con­trol. This is the same point I re­al­ized that I could spec­ify the mi­cro­phone sam­ple rate for the op­ti­miza­tion I could added from the last sec­tion. Most of these op­tions are on by de­fault de­pend­ing on your browser, but I added them ex­plic­itly any­way:

Now that some­what helped, but Nero still picks up it’s own au­dio. This prob­a­bly re­quires a more so­phis­ti­cated so­lu­tion, but I moved on to the next prob­lem.

The sec­ond ob­vi­ous fail­ure mode is the fact that it picks up key­board clicks, and the si­lence time­out is hard to tune. My first at­tempt to fix this was to “ignore” large spikes in au­dio by “smoothing” the vol­ume at each frame:

Then, with some ad­vice from Paulo Valente, I im­ple­mented a bi­quad fil­ter to with a low and high-pass in or­der to fil­ter au­dio to the range of hu­man speech:

In prac­tice, both of these so­lu­tions ac­tu­ally seemed to work de­cent, but they could ab­solutely be bet­ter. I know it’s pos­si­ble to im­prove the client-side fil­ter­ing us­ing a rolling-win­dow that looks en­ergy of the speak­ing fre­quences rel­a­tive to en­ergy of an en­tire sam­ple. But, there are also ma­chine learn­ing mod­els that per­form VAD, and have 1ms in­fer­ence times. I re­al­ized that it’s prob­a­bly quicker to just send all of the data over the web­socket in chunks, and per­form VAD on the server. I’ll dis­cuss that im­ple­men­ta­tion a lit­tle later.

Next I wanted to add sup­port for in­ter­rup­tions. In the Retell ex­am­ple, the speaker will cut off mid-speech if it de­tects that you are speak­ing. To im­ple­ment this fea­ture in Nero, I added a pu­shEvent to the Microphone hook which would push an in­ter­rupt event to the server any­time speech is de­tec­tected:

The server han­dles this event and broad­casts an event to the TTS chan­nel to stop speak­ing:

And the chan­nel han­dles the event by clear­ing out the out­put au­dio stream and queue:

Unfortunately, this does cre­ate a race con­di­tion. There’s a po­ten­tial sit­u­a­tion where a speaker in­ter­rupts and the speak­ing queue gets cleared on the client, but ElevenLabs is still stream­ing au­dio back to the server. The server is al­ways go­ing to just broad­cast this info to the client, and as is the client will process it. This po­ten­tially cre­ates a sit­u­a­tion with weird con­tin­u­ta­tions in the au­dio. To get around this, I refac­tored the TTS im­ple­men­ta­tion so that each au­dio broad­cast ap­pends a 6 digit to­ken to the pay­load. Then, all we need to do is keep the to­ken in sync with the client and server. On the client, when pro­cess­ing the au­dio queue, it sim­ply checks whether or not the to­ken at the be­gin­ning of the pay­load matches, and if it does­n’t it ig­nores that sam­ple.

The lim­i­ta­tion with this im­ple­men­ta­tion is it does not up­date the chat tran­script. It’s en­tirely pos­si­ble be­cause we have ac­cess to the align­ment in­for­ma­tion from ElevenLabs, but I just did­n’t im­ple­ment it at this time.

Another thing the Retell demo has sup­port for is cues and hang ups af­ter a du­ra­tion of si­lence. If you are silent for too long, you’ll get a cue from the AI speaker ask­ing you if you’re still there. After an­other du­ra­tion of si­lence, it will hang up. This is some­thing that’s pretty easy to do with LiveView and Process.send_after/4:

And then we can can­cel the timer any­time we re­ceive a tran­scrip­tion, and restart it af­ter every turn speak­ing. Note that we can’t de­pend on the Phoenix speak async task end­ing as the trig­ger to send nudges. Instead, we need to push an event from the speaker hook that the au­dio has ended. This avoids a case where the speaker ini­ti­ates a re­ally long speech, which over­laps with the nudge_ms du­ra­tion. Now, we can con­trol the num­ber of nudges with an as­sign. In my case, I just used a boolean:

Somewhere along the line I re­al­ized that my at­tempts at en­gi­neer­ing solid VAD client-side were never go­ing to de­liver the ex­pe­ri­ence that I wanted. I dis­cussed with Andres Alejos a bit, and he found a Silero VAD model which is ca­pa­ble of per­form­ing VAD in 1ms on a sin­gle CPU thread. They also had an ONNX model—and we have a li­brary in the Elixir ecosys­tem called Ortex which al­lows us to ex­e­cute ONNX mod­els.

To ac­co­mo­date for the new VAD model, I ended up re-im­ple­ment­ing the orig­i­nal LiveView I had as a WebSocket. This ac­tu­ally works out well be­cause the WebSocket server is generic, and can be con­sumed by any lan­guage with a WebSocket client. The im­ple­men­ta­tion is also rel­a­tively sim­ple, and eas­ily ex­panded to ac­co­mo­date for other LLMs, TTS, and STT mod­els. The WebSocket im­ple­men­ta­tion has low la­tency (when run­ning on a GPU), and sup­ports in­ter­rupts.

You can find the pro­ject on my GitHub as well as an ex­am­ple us­ing the server.

The fi­nal im­ple­men­ta­tion I ended up with still does not match the qual­ity of the Retell demo. That said, I think it’s a solid start for fu­ture work. I be­lieve I acted with some hubris when first post­ing about this pro­ject, and I would like to say that Retell’s work should not be un­der­stated. I can ap­pre­ci­ate the at­ten­tion to de­tail that goes into mak­ing an ef­fec­tive con­ver­sa­tional agent, and Retell’s demo shows they paid a lot of at­ten­tion to the de­tails. Kudos to them and their team.

I will also ad­mit that my demo is play­ing to one bench­mark. I’m op­ti­miz­ing the hell out of la­tency to sup­port a sin­gle user—me. I think this so­lu­tion would change if it needed to ac­co­mo­date for mul­ti­ple con­cur­rent users.

Retell’s web­site claims they have a con­ver­sa­tion or­ches­tra­tion model un­der the hood to man­age the com­plex­i­ties of con­ver­sa­tion. I had my doubts about that go­ing into this, but I be­lieve it now. Whether or not this model is ac­tu­ally a sin­gle model or a se­ries of mod­els for VAD, adding backchan­nels, etc. I’m not sure. I think even­tu­ally it will be a sin­gle model, but I’m not sure if it is now, which leads me to my next point.

While do­ing all of these op­ti­miza­tions, I could not help but think that it will even­tu­ally be all for naught. Not be­cause I don’t think peo­ple will find it use­ful, but be­cause large mod­els trained on lots of data sim­ply seem to al­ways beat en­gi­neer­ing ef­fort. I be­lieve the fu­ture of this area of work is in joint mod­els. I think the only way to achieve real-time con­ver­sa­tions is to merge parts of the stack. I pre­dict in less than a year we will see an in­cred­i­bly ca­pa­ble joint speech/​text model. I re­cently saw a large au­dio model called Qwen-Audio that I be­lieve is sim­i­lar to what I en­vi­sion.

Specifically, if some­body were kind enough to give me some money to throw at this prob­lem, here is ex­actly what I would do:

Generate an Alpaca-style and/​or LLaVA-style dataset of syn­thetic speech. Note that it would re­quire a bit of pre-pro­cess­ing to change Alpaca in­puts to mir­ror a style com­pat­i­ble with spo­ken-word. I would use ElevenLabs to gen­er­ate the dataset in mulit­ple voices. Of course this dataset would be a bit too “clean,” so we’d need to ap­ply some aug­men­ta­tions which add am­bi­ent noise, change speak­ing pitch and speed, etc. Bonus points: adding sam­ples of “noise” which re­quire no re­sponse to merge the VAD part of the pipeline in as well. You can even throw in text prompts that dic­tate when and when not to re­spond to sup­port things like wake word de­tec­tion with­out need­ing to train a sep­a­rate model.

Create a LLaVA-style model with a Whisper or equiv­a­lent base, an LLM, and a pro­jec­tion layer.

Secure H100s, train model, and “turn H100s into $100s” (thank you @thmsmlr)

If you want to give me some $$$, my e-mail is smo­ri­ar­ity.5@gmail.com 🙂

I be­lieve we are also close to just hav­ing full-on speech-to-speech mod­els. A spe­cific chal­lenge I can see when cre­at­ing these mod­els is com­ing up with a high-qual­ity dataset. I think if you make a de­lib­er­ate at­tempt at “recording con­ver­sa­tions” for the pur­poses of train­ing, you will ac­tu­ally prob­a­bly end up with a lower-qual­ity dataset. People tend to change their be­hav­ior un­der ob­ser­va­tion. Additionally, con­ver­sa­tions from movies and TV shows aren’t ac­tu­ally very nat­ural. Even some pod­casts have an un­nat­ural con­veras­tional rhythm.

While watch­ing Love is Blind with my fi­ancé, I re­al­ized you could prob­a­bly get a de­cent amount of qual­ity data from re­al­ity tv shows. The con­ver­sa­tions in re­al­ity TV are overly dra­matic and chaotic, but are (I think) closer to re­al­is­tic than any­thing else.

I do won­der what a solid RAG im­ple­men­ta­tion looks like on top of a con­ver­sa­tional agent. RAG and com­plex CoT pipelines will in­tro­duce la­tency which could de­te­ri­o­rate the con­ver­sa­tional ex­pe­ri­ence. However, there are clever ways you can hide this. In con­ver­sa­tions that re­quire “search” be­tween hu­mans, e.g. like sched­ul­ing an ap­point­ment, you’ll of­ten have one party say­ing “one mo­ment please” be­fore per­form­ing a sys­tem search. Building some­thing like that in is en­tirely pos­si­ble. Additionally, if your agent re­quires in­for­ma­tion up front about an in­di­vid­ual, it’s pos­si­ble to in­clude that in the ini­tial prompt.

I was very ex­cited for this prob­lem in par­tic­u­lar be­cause it’s lit­er­ally the per­fect ap­pli­ca­tion of Elixir and Phoenix. If you are build­ing con­ver­sa­tional agents, you should se­ri­ously con­sider giv­ing Elixir a try. A large part of how quick this demo was to put to­gether is be­cause of how pro­duc­tive Elixir is.

This was a fun tech­ni­cal chal­lenge. I am pleased with the per­for­mance of the fi­nal demon­stra­tion. I’m also happy I was able to OSS a small li­brary for oth­ers to build off of. If you are in­ter­ested in con­ver­sa­tional agents, I en­cour­age you to check it out, give feed­back, and con­tribute! I know it’s very rough right now, but it will get bet­ter with time.

Additionally, I plan to pe­ri­od­i­cally build out the rest of the Nero pro­ject, so please fol­low me on Twitter if you’d like to stay up to date.

Every JRMF puz­zle is hands-on, play-based, and stan­dards-aligned.

We de­sign our ac­tiv­i­ties to have a low floor so that any­one can find a way to en­gage and a high ceil­ing so that every­one can find a mean­ing­ful chal­lenge.

All of our puz­zles come with free fes­ti­val guides that help you use our ac­tiv­i­ties at home, in the class­room, or dur­ing math fes­ti­vals.

Can you be the first player to get 3-in-a-row?

After a long day of pick­ing ap­ples, can you eat the last, juici­est ap­ple?

Embrace your in­ner pool shark and try to make every shot.

Can you re­move all the dots?

Can you build a bridge that con­nects the stars?

Can you make the longest line?

Help all the chameleons on the is­land change into the same color.

Can you make the per­fect choco­late box for your cus­tomers?

Can you avoid tak­ing the yucky choco­late piece?

Create col­or­ful loops so that you can al­ways find your way back to the be­gin­ning of the path.

Be the player to take the last to­ken!

Crack the se­cret code in as few guesses as you can!

Can you find a way to stack every cup?

Use every digit of your num­bers to solve these unique and chal­leng­ing num­ber puz­zles.

Can you tile the board us­ing each domino ex­actly once?

Can you trace each doo­dle with­out lift­ing your marker?

Can you make a bet­ter die than your part­ner?

Can you help the frogs and toads get to the other side of the pond?

Can you split up each puz­zle so that more than half of the groups are pur­ple groups?

Can you grace­fully la­bel these pump­kin patches?

Learn the mys­ter­ies of the sim­ple yet sur­pris­ing hexa­flexagon, and take one home to show your friends!

Can you jump your way from start to fin­ish?

Everything is made up of one, con­tin­u­ous line. Can you fig­ure out what be­longs in town?

Help as many la­dy­bugs as pos­si­ble land on the leaves.

Fill your en­tire gar­den with plots of car­rots.

Can you arrange 5 num­bers into a Magic Flower in which every group of three num­bers in a line adds up to the same magic num­ber?

Can you color each map so that no two neigh­bor­ing states are col­ored the same color?

Can you find all the places where the Meeple can live?

Race your friends to find the pat­terns in these Halloween-themed SET cards!

Can you make each mo­saic?

Can you make palin­dromes in the fewest num­ber of swaps?

Among all the coun­ter­feits there is only one true gold bar. Can you find it?

Can you find a way to cover the shapes with pen­tomi­noes?

Can you build a poly­he­dron us­ing the shapes in each puz­zle?

Learn how doc­tors help pa­tients us­ing pool test­ing through this en­gag­ing game.

Armed with only your un­der­stand­ing of prime num­bers, play against your friends to find the best way to fill in our col­or­ful cube.

Help your wolf, goat, and cab­bage cross the river.

Race your friends to get the rook to the end of the chess­board. Can you find a way to win every time?

Can you make the pat­tern in the puz­zle by ro­tat­ing the cube?

Help the city plan­ner build up the city with new sky­scrap­ers.

Smiling is con­ta­gious! Can you turn every frown up­side down?

Take turns draw­ing curves to con­nect dots. Can you be the one to draw the last curve?

Can you place the stars on the grid?

Can you find a way to place the next step­ping stone?

Can you cover the en­tire square with pat­tern blocks?

How many toads can you help sun­bathe by bal­anc­ing them on a lily­pad?

Can you make the right num­ber of squares out of tooth­picks?

Can you make the right num­ber of tri­an­gles out of tooth­picks?

Can you solve these puz­zles based on an an­cient leg­end, golden disks, and the end of the world?

Keep your sheep safe from the wolves!

Google has fired 28 em­ploy­ees over their par­tic­i­pa­tion in a 10-hour sit-in at the search gi­ant’s of­fices in New York and Sunnyvale, California, to protest the com­pa­ny’s busi­ness ties with the Israel gov­ern­ment, The Post has learned.

The pro-Pales­tin­ian staffers — who had donned tra­di­tional Arab head­scarves as they stormed and oc­cu­pied the of­fice of a top ex­ec­u­tive in California on Tuesday — were ter­mi­nated late Wednesday af­ter an in­ter­nal in­ves­ti­ga­tion, Google vice pres­i­dent of global se­cu­rity Chris Rackow said in a com­pa­ny­wide memo.

“They took over of­fice spaces, de­faced our prop­erty, and phys­i­cally im­peded the work of other Googlers,” Rackow wrote in the memo ob­tained by The Post. “Their be­hav­ior was un­ac­cept­able, ex­tremely dis­rup­tive, and made co-work­ers feel threat­ened.”

In New York, pro­test­ers had oc­cu­pied the 10th floor of Google’s of­fices in the Chelsea sec­tion of Manhattan as part of a protest that also ex­tended to the com­pa­ny’s of­fices in Seattle for what it called “No Tech for Genocide Day of Action.”

“Behavior like this has no place in our work­place and we will not tol­er­ate it,” Rackow wrote. “It clearly vi­o­lates mul­ti­ple poli­cies that all em­ploy­ees must ad­here to — in­clud­ing our code of con­duct and pol­icy on ha­rass­ment, dis­crim­i­na­tion, re­tal­i­a­tion, stan­dards of con­duct, and work­place con­cerns.”

Rackow added that the com­pany “takes this ex­tremely se­ri­ously, and we will con­tinue to ap­ply our long­stand­ing poli­cies to take ac­tion against dis­rup­tive be­hav­ior — up to and in­clud­ing ter­mi­na­tion.”

The fired staffers are af­fil­i­ated with a group called No Tech For Apartheid, which has been crit­i­cal of Google’s re­sponse to the Israel-Hamas war.

The group had posted sev­eral videos and livestreams of the protests on its X ac­count — in­clud­ing the ex­act mo­ment that em­ploy­ees were is­sued fi­nal warn­ings and ar­rested by lo­cal po­lice for tres­pass­ing.

The pro­test­ers have de­manded that Google pull out of a $1.2 bil­lion “Project Nimbus” con­tract — in which Google Cloud and Amazon Web Services pro­vide cloud-com­put­ing and ar­ti­fi­cial in­tel­li­gence ser­vices for the Israeli gov­ern­ment and mil­i­tary.

Critics at the com­pany raised con­cerns that the tech­nol­ogy would be weaponized against Palestinians in Gaza.

The im­pacted work­ers blasted Google over the fir­ings in a state­ment shared by No Tech For Apartheid spokesper­son Jane Chung.

“This evening, Google in­dis­crim­i­nately fired 28 work­ers, in­clud­ing those among us who did not di­rectly par­tic­i­pate in yes­ter­day’s his­toric, bi­coastal 10-hour sit-in protests,” the work­ers said in the state­ment.

“This fla­grant act of re­tal­i­a­tion is a clear in­di­ca­tion that Google val­ues its $1.2 bil­lion con­tract with the geno­ci­dal Israeli gov­ern­ment and mil­i­tary more than its own work­ers — the ones who cre­ate real value for ex­ec­u­tives and share­hold­ers.”

“Sundar Pichai and Thomas Kurian are geno­cide prof­i­teers,” the state­ment added, re­fer­ring to Google’s CEO and the CEO of its cloud unit, re­spec­tively.

“We can­not com­pre­hend how these men are able to sleep at night while their tech has en­abled 100,000 Palestinians killed, re­ported miss­ing, or wounded in the last six months of Israel’s geno­cide — and count­ing.”

An NYPD spokesper­son said the Tuesday protest “involved ap­prox­i­mately 50 par­tic­i­pants” in to­tal and con­firmed “four ar­rests were made for tres­pass­ing in­side the Google build­ing.”

The Sunnyvale Department of Public Safety said the protest in California “consisted of around 80 par­tic­i­pants.” A to­tal of five pro­test­ers who re­fused to leave the Google of­fice were “arrested with­out in­ci­dent for crim­i­nal tres­pass­ing,” booked and re­leased, a spokesper­son added.

It could­n’t im­me­di­ately be learned if all nine ar­rested em­ploy­ees were among those who were fired. Google had ear­lier placed the em­ploy­ees on ad­min­is­tra­tive leave and cut their ac­cess to in­ter­nal sys­tems.

Last month, Google fired a soft­ware en­gi­neer who pub­licly blasted one of the com­pa­ny’s Israel-based ex­ec­u­tives dur­ing a tech con­fer­ence in New York City.

When reached for com­ment, a Google spokesper­son con­firmed the fir­ings.

“These protests were part of a long­stand­ing cam­paign by a group of or­ga­ni­za­tions and peo­ple who largely don’t work at Google,” the spokesper­son said in a state­ment.

“A small num­ber of em­ployee pro­test­ers en­tered and dis­rupted a few of our lo­ca­tions. Physically im­ped­ing other em­ploy­ees’ work and pre­vent­ing them from ac­cess­ing our fa­cil­i­ties is a clear vi­o­la­tion of our poli­cies, and com­pletely un­ac­cept­able be­hav­ior.”

“We have so far con­cluded in­di­vid­ual in­ves­ti­ga­tions that re­sulted in the ter­mi­na­tion of em­ploy­ment for 28 em­ploy­ees, and will con­tinue to in­ves­ti­gate and take ac­tion as needed,” the spokesper­son added.

The demon­stra­tors stormed the per­sonal of­fice of Google Cloud CEO Thomas Kurian in Sunnyvale.

Kurian’s cus­tom-made, framed Golden State Warriors jer­sey was vis­i­ble on the of­fice wall in the back­ground of the livestream, and em­ploy­ees wrote a list of their de­mands on his white board.

The com­pa­ny­wide memo can be read in its en­tirety be­low.

You may have seen re­ports of protests at some of our of­fices yes­ter­day. Unfortunately, a num­ber of em­ploy­ees brought the event into our build­ings in New York and Sunnyvale. They took over of­fice spaces, de­faced our prop­erty, and phys­i­cally im­peded the work of other Googlers. Their be­hav­ior was un­ac­cept­able, ex­tremely dis­rup­tive, and made co-work­ers feel threat­ened. We placed em­ploy­ees in­volved un­der in­ves­ti­ga­tion and cut their ac­cess to our sys­tems. Those who re­fused to leave were ar­rested by law en­force­ment and re­moved from our of­fices.

Following in­ves­ti­ga­tion, to­day we ter­mi­nated the em­ploy­ment of twenty-eight em­ploy­ees found to be in­volved. We will con­tinue to in­ves­ti­gate and take ac­tion as needed.

Behavior like this has no place in our work­place and we will not tol­er­ate it. It clearly vi­o­lates mul­ti­ple poli­cies that all em­ploy­ees must ad­here to — in­clud­ing our Code of Conduct and Policy on Harassment, Discrimination, Retaliation, Standards of Conduct, and Workplace Concerns.

We are a place of busi­ness and every Googler is ex­pected to read our poli­cies and ap­ply them to how they con­duct them­selves and com­mu­ni­cate in our work­place. The over­whelm­ing ma­jor­ity of our em­ploy­ees do the right thing. If you’re one of the few who are tempted to think we’re go­ing to over­look con­duct that vi­o­lates our poli­cies, think again. The com­pany takes this ex­tremely se­ri­ously, and we will con­tinue to ap­ply our long­stand­ing poli­cies to take ac­tion against dis­rup­tive be­hav­ior — up to and in­clud­ing ter­mi­na­tion.

You should ex­pect to hear more from lead­ers about stan­dards of be­hav­ior and dis­course in the work­place.