KAISER PERMANENTE NURSES who an­swer ad­vice and triage calls say their duty of care for pa­tients is be­ing in­creas­ingly threat­ened by work­place sur­veil­lance.

Seven cur­rent and for­mer nurses told CalMatters that those who spend more than 15 min­utes on a call with a pa­tient rou­tinely face crit­i­cism from Kaiser man­age­ment or get called into per­for­mance eval­u­a­tion meet­ings. Call time, they said, fac­tors into monthly per­for­mance scores they re­ceive.

In ad­di­tion to track­ing call length, they said Kaiser uses soft­ware that tries to pre­dict on a daily ba­sis whether they’re be­ing un­pro­duc­tive or fail­ing to an­swer calls quickly. Artificial in­tel­li­gence sys­tems have also been used to rate their em­pa­thy and tone of voice.

Their com­ments come as the California Nurses Association be­gins ne­go­ti­at­ing a new con­tract with Kaiser this month with AI a likely is­sue. Kaiser nurses went on strike against AI for one day in March and pick­eted against AI last fall. The CNA is bar­gain­ing for 25,000 nurses, in­clud­ing 1,000 in call cen­ters.

At the same time, California law­mak­ers are con­sid­er­ing sev­eral bills reg­u­lat­ing AI in the work­place, in­clud­ing one that would pro­tect from re­tal­i­a­tion doc­tors and nurses who over­ride au­to­mated care rec­om­men­da­tions.

Kaiser de­fended its use of AI, say­ing it de­ploys the tech­nol­ogy with pa­tient safety in mind and does not use “average han­dle time” to as­sess per­for­mance.

Kaiser Permanente is the largest pri­vate em­ployer in California, pro­vid­ing health­care ser­vices to more than 9 mil­lion peo­ple in the state and to 3 mil­lion other Americans. That means the com­pa­ny’s use of ar­ti­fi­cial in­tel­li­gence could set im­por­tant prece­dents for man­ag­ing work­ers with AI. It could also have a big im­pact on pa­tient care, pro­vid­ing an early ex­am­ple of how the health­care sec­tor bal­ances cost-cut­ting au­toma­tion with hu­man pres­ence or touch.

Raquel Alvarez Sanchez, a Kaiser Permanente ad­vice nurse in Vallejo since 2010, said she was on a call with a pa­tient who was sui­ci­dal last year that took more than an hour be­cause she had to wait for po­lice to ar­rive be­fore hang­ing up. She tried to make the man feel cared for, even though she was cog­nizant that stay­ing on the call that long would throw off her av­er­age call time for weeks and could lead to ques­tions from man­age­ment. Sanchez, a union stew­ard, said she’s ac­com­pa­nied col­leagues to per­for­mance eval­u­a­tion meet­ings, where they were found to have done every­thing right on a call — ex­cept stay­ing on the line for more than 15 min­utes. She said she has­n’t seen nurses get fired for do­ing that, but she fears that con­tin­ued pres­sure can lead nurses to quit or re­tire early.

“I think at some point all of the nurses have been talked to about their av­er­age han­dle time,” she said. “The only thing I can think of is they’re do­ing it for profit.”

Another nurse who spoke with CalMatters on con­di­tion of anonymity due to fear of ret­ri­bu­tion de­scribed how that sur­veil­lance af­fected a call with a pa­tient last year. Initially she thought her pa­tient, an el­derly woman who just re­ceived a ter­mi­nal can­cer di­ag­no­sis, was sui­ci­dal, but quickly came to un­der­stand that she was in shock and re­ally needed some­body to talk to.

The nurse wanted to take time to show com­pas­sion or com­fort to the woman, who acts as a care­taker for her daugh­ter, but she stopped her­self out of fear it would hurt her monthly per­for­mance score and lead to a rep­ri­mand from her man­ager. She be­came a nurse to pro­vide peo­ple with com­pas­sion­ate care, but “I had to ask my­self: Am I go­ing to get dis­ci­plined for go­ing off script or say­ing more than what is nec­es­sary?”

Kaiser Permanente says its per­for­mance eval­u­a­tions help im­prove pa­tient out­comes. A com­pany spokesper­son said, “Kaiser Permanente does not use Average Handle Time to as­sess agent per­for­mance or en­force call time met­rics. Any tools used in con­tact cen­ter set­tings sup­port our qual­ity as­sur­ance ef­forts and have hu­man re­view and over­sight.” In a state­ment pro­vided to CalMatters, spokesper­son Vincent Staupe added that Kaiser uses AI re­spon­si­bly, with hu­man over­sight, and by “prioritizing pa­tient safety, pri­vacy, and eq­uity,” but he said, “As a large or­ga­ni­za­tion, we do not share spe­cific in­for­ma­tion about in­ter­nal tech­nol­ogy sys­tems for se­cu­rity and op­er­a­tional rea­sons.”

Is tech­nol­ogy putting pa­tients at risk?

It’s not clear how pa­tient care is af­fected by al­go­rith­mic man­age­ment, nor is the im­pact of lim­it­ing the length of triage and ad­vice calls on pa­tients. Kaiser call cen­ter nurses can’t say for cer­tain whether the pres­sures they face re­sults in ad­verse out­comes for pa­tients be­cause their con­tact with pa­tients ends af­ter they hang up the phone. A 2024 pub­lic records re­quest by CalMatters to the California Department of Managed Health Care found no com­plaints by pa­tients against Kaiser re­lated to call times. But nurses in­sist the risk to pa­tient safety and qual­ity of care is real.

Consumer Watchdog pa­tient ad­vo­cate Michele Ramos said many Kaiser pa­tients be­gin their care on the ad­vice line. They later com­plain to her, mostly about things that hap­pen in Kaiser fa­cil­i­ties, but “I can see now where a lot of the prob­lems” start, given the call con­straints nurses are un­der.

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, … which is only go­ing to fail pa­tients.”

Michele Ramos, Consumer Watchdog pa­tient ad­vo­cate

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, … which is only go­ing to fail pa­tients.”

Michele Ramos, Consumer Watchdog pa­tient ad­vo­cate

Ramos said the time pres­sures may fit a broader pat­tern at Kaiser of putting costs over qual­ity. The health gi­ant was hit with a record fine, $50 mil­lion, as part of a set­tle­ment over find­ings from the California Department of Managed Health Care that it de­layed be­hav­ioral health ap­point­ments be­yond statu­tory lim­its and too of­ten moved pa­tients into group rather than in­di­vid­ual ther­apy. Kaiser also set­tled with the U.S. Department of Labor after in­ves­ti­ga­tions into its sub­stance use and men­tal health ser­vices. Kaiser faced crit­i­cism in 2002 for pay­ing bonuses to call cen­ter work­ers who aren’t nurses for keep­ing calls short, though call cen­ter nurses who spoke with CalMatters said they en­coun­tered no such prac­tices to­day.

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, and I think this would be a con­trib­u­tor to that, which is only go­ing to fail pa­tients,” Ramos added.

Nurses said they are pres­sured to stay un­der 15 min­utes even for the sorts of calls that of­ten take more time, like di­ag­nos­ing a pa­tient with mul­ti­ple symp­toms, chronic ill­nesses, new par­ents in need of ad­vice and as­sur­ance, peo­ple who de­sire ex­tended health ed­u­ca­tion, or peo­ple who are over­whelmed af­ter re­ceiv­ing life-al­ter­ing news who could use some com­pas­sion. Nurses say calls that in­volve in­ter­preters of­ten take 30 min­utes or more. About four in 10 Californians speak a lan­guage other than English and half of them do not speak English well, ac­cord­ing to a state en­vi­ron­men­tal health agency.

“The amount of time that Kaiser is giv­ing us to com­plete a call is some­times not safe,” said one nurse, who asked to re­main anony­mous due to fear of re­tal­i­a­tion.

“People can get hurt,” said Charlotte Capulong, who has worked in nurse call cen­ters for 22 years and helped or­ga­nize Kaiser nurses against the AI tone-of-voice tool. Capulong said nurses felt ha­rassed by man­agers in meet­ings she at­tended as a union rep, even if they suc­cess­fully car­ried out all other du­ties of their jobs ex­cept com­plet­ing calls within 15 min­utes.

“You aren’t call­ing Comcast. We’re deal­ing with life here,” she said.

Nurses are in­structed to stick to a script on phone calls and give no more than two to three pieces of ad­vice, Capulong and other nurses said, which means they may some­times need to de­cide whether to with­hold ad­vice or face a per­for­mance eval­u­a­tion hear­ing.

The nurses say ar­ti­fi­cial in­tel­li­gence could make the sur­veil­lance nurses en­counter on the job worse.

In sum­mer 2024, Kaiser be­gan test­ing an AI tool that at­tempts to as­sess em­pa­thy and tone in the voices of nurses and pa­tients, ac­cord­ing to nurses who spoke with CalMatters. In re­sponse, nurses cir­cu­lated and signed a pe­ti­tion in fa­vor of the right to pa­tient pri­vacy, more trans­parency,  and the right to ex­er­cise their pro­fes­sional judge­ment and en­cour­aged man­age­ment to in­volve nurse’s in­put and feed­back. The sig­na­ture cam­paign used the same tag line that nurses used at protests out­side San Francisco hos­pi­tals ear­lier that year: “Trust nurses, not AI. The AI tests ended in November 2024, but union rep­re­sen­ta­tives were told that man­agers may bring the pro­gram back in the fu­ture.

Nurses re­ported feel­ing ha­rassed by ex­ist­ing sur­veil­lance, “and that was in­ten­si­fied when they said we’re go­ing to use AI to eval­u­ate our calls and grade us,” said Sanchez.

Another nurse speak­ing on con­di­tion of anonymity said “AI did not un­der­stand our job and would grade us wrong all the time.”

A Kaiser spokesper­son de­clined to re­spond to ques­tions about the AI tool or an­swer ques­tions about the use of AI and other au­to­mated sys­tems in the com­pa­ny’s call cen­ters and health­care fa­cil­i­ties, in­clud­ing for eval­u­at­ing nurse per­for­mance or whether pa­tients were in­formed about the use of AI to eval­u­ate their em­pa­thy and tone.

Nurses also said they get lit­tle time be­tween calls even if that call in­volves speak­ing with a pa­tient who is sui­ci­dal, ex­pe­ri­enc­ing a men­tal health episode, or near death. In years past, nurses got around 10 min­utes to fin­ish writ­ing notes in a pa­tien­t’s chart or col­lect them­selves af­ter a par­tic­u­larly tough call. Today they say they typ­i­cally get 30 sec­onds or less when lines are busy, al­though more at slow times, like late at night, or if they get a man­ager’s per­mis­sion af­ter a par­tic­u­larly chal­leng­ing call. The over­all pace they say, can lead to mis­takes like miss­ing im­por­tant cues into a pa­tien­t’s well­be­ing.

CNA reps de­clined to talk about spe­cific pro­vi­sions they in­tend to seek re­lated to AI ahead of their talks with Kaiser this sum­mer.

How sur­veil­lance and AI shape nurs­ing

Critics say ex­ces­sive work­place mon­i­tor­ing can lead to lower morale as em­ploy­ees feel less trusted and au­tonomous, rel­e­gated to be­ing no more than al­go­rithm mon­i­tors. UC Berkeley Labor Center Technology and Work Program di­rec­tor An­nette Bernhardt has warned that al­go­rith­mic man­age­ment can turn peo­ple into fleshy ro­bots, echo­ing com­plaints from an Ama­zon fac­tory worker who CalMatters in­ter­viewed last year. A 2023 aca­d­e­mic sur­vey of call cen­ters in four de­vel­oped coun­tries found that us­ing AI for man­age­ment or mon­i­tor­ing left work­ers with less time be­tween calls and more likely to feel emo­tion­ally drained by their work. Nearly half of re­spon­dents said that AI tools made their jobs more stress­ful. A prior study by the same re­searchers, Virginia Dolleghast of Cornell University and Sean O’Brady of McMaster University found that per­for­mance mon­i­tor­ing leads to higher rates of emo­tional ex­haus­tion.

Dolleghast, who has stud­ied the im­pact of sur­veil­lance tech­nol­ogy on call cen­ter work­ers for more than a decade, said what Kaiser call cen­ter nurses are ex­pe­ri­enc­ing is part of a broader trend: Across dif­fer­ent in­dus­tries, per­sis­tent sur­veil­lance is in­creas­ing stress lev­els for work­ers who are re­solv­ing com­plex, emo­tion­ally-charged is­sues.

“Stress and burnout can lead to more mis­takes across a range of ar­eas, and in the health­care set­ting that is much higher risk be­cause you’re deal­ing with peo­ple’s lives and their health,” she said.

The con­verse can be true: Workers who are given more dis­cre­tion to de­cide the pace and tim­ing of their work ex­pe­ri­ence higher lev­els of job sat­is­fac­tion and less ab­sen­teeism.

Nurses na­tion­wide are more fre­quently en­coun­ter­ing ar­ti­fi­cial in­tel­li­gence and sim­i­lar soft­ware sys­tems in the work­place. Half of more than 2,000 nurses who re­sponded to a 2024 sur­vey by the National Nurses United union said their em­ployer uses al­go­rith­mic sys­tems to an­a­lyze health records. Such sys­tems can do things like de­ter­mine how frag­ile a pa­tient is or pre­dict how many hours of care they will need. Two-thirds of the sur­veyed nurses said their own as­sess­ments had at some point dis­agreed with a com­puter-gen­er­ated pre­dic­tion. Six out of 10 re­spon­dents said they don’t trust their em­ployer to pri­or­i­tize pa­tient safety when us­ing AI.

Pa Vue has worked as a nurse in call cen­ters for the bet­ter part of the past decade. She said she and other Kaiser nurses rou­tinely have con­ver­sa­tions with man­agers about call ef­fi­ciency and re­ceive eval­u­a­tion scores once a month. She re­calls hav­ing a score re­duced for re­peat­ing ad­vice to a pa­tient that she wor­ried had un­usual symp­toms and pos­si­ble heart is­sues.

As a union rep­re­sen­ta­tive in some per­for­mance meet­ings, Vue has seen man­agers raise ef­fi­ciency ques­tions about calls they deem too long. She’s also seen nurses re­ceive lower per­for­mance scores if they go against soft­ware rec­om­men­da­tions based on their pro­fes­sional opin­ion or make an ap­point­ment for a pa­tient with­out con­sult­ing a doc­tor.

She be­lieves that ef­fi­ciency aims ac­cel­er­ated by tech­nol­ogy can hin­der a nurse’s abil­ity to fo­cus and re­duce the qual­ity of care that pa­tients pay for.

“I’m not against the use of AI as long as it’s ben­e­fi­cial to the pa­tient but in this par­tic­u­lar use [empathy and tone mon­i­tor­ing] it’s to in­crease pro­duc­tiv­ity and im­prove ef­fi­ciency and cut costs. Kaiser is for­get­ting we aren’t just a call cen­ter for cus­tomer sup­port, we’re nurses, and we’re there to take care of pa­tients,” she said.

As AI im­proves and busi­nesses push work­ers to use it, unions are, in turn, in­creas­ingly de­mand­ing that em­ploy­ers ad­dress is­sues raised by AI when bar­gain­ing for new con­tracts. Surveillance tech­nol­ogy has be­come a com­mon way for man­agers to col­lect data about work­ers in a num­ber of in­dus­tries, used for every­thing from im­prov­ing safety to hunt­ing for ways to in­crease profit gains or train AI to do a job.

At Kaiser, AI is a key is­sue not only among nurses but also for men­tal health work­ers, 2,400 of whom are in con­tract ne­go­ti­a­tions in Northern California with Kaiser Permanente. Kaiser ther­a­pists have said they are con­cerned about use of ther­apy ses­sion tran­scripts to train AI mod­els and about the health-care gi­ant us­ing AI to take their jobs. National Union of Healthcare Workers spokesper­son Matt Artz told CalMatters con­tract ne­go­ti­a­tions are on­go­ing.

How Kaiser uses AI

Kaiser Permanente is ex­plor­ing or us­ing AI in many parts of the health­care ex­pe­ri­ence far be­yond nurse call cen­ters. Kaiser uses AI to iden­tify pa­tients in hos­pi­tals at risk of ad­verse events by eval­u­at­ing data on their elec­tronic health records. An AI sys­tem called Preventus is used to de­ter­mine when to dis­charge pa­tients. Doctors and ther­a­pists use Abridge to record in­ter­ac­tions and trans­late speech to text dur­ing in-per­son vis­its with pa­tients in­stead of tak­ing notes. Remote mon­i­tor­ing with AI for pa­tients that need ex­tra care has been tested at Kaiser Permanente fa­cil­i­ties in the Bay Area, ac­cord­ing to nurses who en­coun­tered the tech­nol­ogy in the course of do­ing their jobs.

National Nurses United and CNA President Cathy Kennedy sees the use of AI to de­tect nurse em­pa­thy as part of a long se­ries of steps by Kaiser to limit their au­ton­omy and make them more ef­fi­cient. She be­lieves AI threat­ens to au­to­mate and frag­ment the work that nurses do, and com­pa­nies de­vel­op­ing and de­ploy­ing AI sys­tems should es­tab­lish that those sys­tems are ef­fec­tive and eq­ui­table be­fore de­ploy­ing them.

Notification of new tech de­ploy­ments is part of the nurse union’s con­tract with Kaiser but some­times nurses don’t re­ceive no­ti­fi­ca­tion, CNA says. So union lead­ers are at­tempt­ing to track the num­ber of AI mod­els in use at Kaiser Permanente, ad­vis­ing its mem­bers to in­form them when they en­counter new tech. This paves the way for CNA to push back as it did with the em­pa­thy and tone AI last sum­mer or as it did when it stopped a pi­lot pro­gram that would have re­placed nurses that sit at the bed­side of con­fused pa­tients with cam­eras.

Debru Carthan, a Kaiser ra­di­ol­o­gist, is on the front line of worker-man­age­ment fights over AI at the com­pany. A mem­ber of Service Employees International Union, she is also part of the Coalition of Kaiser Permanente Unions, where she sits on a com­mit­tee to dis­cuss use of AI and emerg­ing tech­nol­ogy at Kaiser. The coali­tion also has a “see some­thing, say some­thing,” cam­paign for front­line work­ers to re­port when they no­tice AI de­ploy­ments; the coali­tion says that too of­ten man­age­ment qui­etly im­ple­ments AI into work­flows with­out no­tice or worker in­put. She wor­ries that the AI tone de­tec­tor used on ad­vice nurses could dis­crim­i­nate against nurses from dif­fer­ent cul­tures and has come to be­lieve that the use of AI in health­care gen­er­ally has more to do with money and cor­po­rate greed than pa­tient care.

California law­mak­ers have re­sponded to worker AI con­cerns both in­side and out­side the health­care sec­tor. They tried and failed last year to ad­dress how AI im­pacts work­ers like call cen­ter nurses. As­sem­bly Bill 1018 and Senate Bill 7, two bills en­dorsed by the CNA, would have re­quired em­ploy­ers to in­form work­ers be­fore us­ing au­to­mated sys­tems on the job to do things like pro­mote or dis­ci­pline work­ers or eval­u­ate job per­for­mance, but Gov. Gavin Newsom ve­toed SB 7, and, fac­ing strong op­po­si­tion from com­pa­nies in­clud­ing Kaiser Permanente, AB 1018 failed to pass for the third con­sec­u­tive year.

Earlier this year, law­mak­ers rein­tro­duced a new ver­sion of Senate Bill 7, now called Senate Bill 947. Another bill would pro­hibit em­ploy­ers us­ing AI to pre­dict the emo­tional state of their em­ploy­ees. Yet another bill would pro­tect doc­tors and nurses from re­tal­i­a­tion if they over­ride rec­om­men­da­tions gen­er­ated by an au­to­mated sys­tem and re­quire health­care providers to sup­ply em­ploy­ees with an in­ven­tory of au­to­mated sys­tems once a year. Kaiser de­clined to share a com­pre­hen­sive list of AI sys­tems in use when asked by CalMatters.

Altogether CNA and the af­fil­i­ated California Labor Federation sup­port roughly half a dozen bills to reg­u­late use of AI in the work­place. Calling AI a cen­tral is­sue in the next pres­i­den­tial elec­tion, mem­bers of the California Labor Federation and la­bor lead­ers from Democratic pri­mary states held a press con­fer­ence in Sacramento ear­lier this year to say that if Newsom wants to be­come pres­i­dent then he needs to pass laws pro­tect­ing work­ers from AI. “It’s an on­go­ing fight, and it’s a fight well worth hav­ing,” Kennedy said. “Whenever there are other unions in dis­cus­sion about ar­ti­fi­cial in­tel­li­gence we are in sol­i­dar­ity with them.”

The nurse that with­held com­pas­sion to a ter­mi­nal can­cer pa­tient she thought was sui­ci­dal said she be­lieves mon­i­tor­ing and scor­ing sys­tems turn nurses into au­toma­tons that check boxes.

“I used to use hu­mor as a way to help pa­tients heal, and I don’t feel com­fort­able do­ing that here be­cause I know the calls are be­ing recorded. You can al­ways tell when a pa­tient ap­pre­ci­ates the hu­mor or your per­sonal com­pas­sion, but I don’t feel like call cen­ters have tol­er­ance for that be­cause that’s not part of the script,” she said. “That re­ally takes away from the whole point of be­ing a nurse and what pa­tients come to know from nurses.”

This story was re­ported with con­tri­bu­tions from Lam Thuy Vo and Ana Ibarra.

This story orig­i­nally ap­peared in CalMatters.

First at­mos­phere found on Earth-like planet in hab­it­able zone of dis­tant star

17 hours ago

Pallab GhoshScience Correspondent

Melissa Weiss/Center for Astrophysics |Harvard & Smithsonian

Researchers have found the first at­mos­phere sur­round­ing an Earth-like, rocky planet or­bit­ing within the hab­it­able zone of a dis­tant star.

The re­searchers say that their dis­cov­ery pro­vides the strongest ev­i­dence yet that worlds with con­di­tions sim­i­lar to Earth could ex­ist be­yond our so­lar sys­tem.

The gas de­tected in the at­mos­phere is he­lium, which would not be able to sup­port life, but other gasses may also be pre­sent.

The lead au­thor, Dr Collin Cherubim of Harvard University, de­scribed the dis­cov­ery as “a big deal”.

“This is the first time any­one has found an at­mos­phere on a rocky planet in the hab­it­able zone of an­other star.”

The planet, called LHS 1140 b, is 48 light-years from Earth or­bit­ing a red star much smaller and cooler than our Sun.

More than 6,000 worlds have been dis­cov­ered or­bit­ing dis­tant stars. But the new dis­cov­ery is sig­nif­i­cant be­cause it brings us a step closer to one of the biggest prizes in sci­ence: the dis­cov­ery of life on an­other world.

The re­searchers, writ­ing in the jour­nal Science, are clear — they have not done that, at least not yet. But for a planet to sup­port life it has to have wa­ter and for that it has to be the right dis­tance from its star: not too close be­cause it will be too hot and not too far, be­cause it will be too cold — but some­where in be­tween where it will be “just right”.

Planetary sci­en­tists call this the “Goldilocks zone”, af­ter the fairy tale girl who was fussy about the tem­per­a­ture of her por­ridge.

Hundreds of plan­ets have been found in the Goldilocks zones of their re­spec­tive stars — but only a few dozen are small and rocky — like our own Earth — which is an­other tick for a plan­et’s abil­ity to sup­port life.

But none of those have been found to have an at­mos­phere.

Until now.

But the only gas dis­cov­ered in the at­mos­phere so far is he­lium, prob­a­bly in the up­per at­mos­phere, which on its own would not sup­port life.

But there may be other, more life-sus­tain­ing gases, lower down. Dr David Charbonneau, also from Harvard, said that the im­por­tant thing was the dis­cov­ery of an Earth-like planet out­side of our so­lar sys­tem with an at­mos­phere.

“People are gen­er­ally in­ter­ested in the big ques­tions: Are we alone? Is there life be­yond the Earth or be­yond our so­lar sys­tem? To that end, this study re­veals the first at­mos­phere dis­cov­ered on a rocky planet in the hab­it­able zone of a star out­side of our so­lar sys­tem,” he said.

LHS 1140b is­n’t the only world un­der scrutiny in the search for life. K2 – 18b, a sub-Nep­tune with a pos­si­ble wa­ter-rich in­te­rior, made head­lines when sci­en­tists spot­ted signs of di­methyl sul­phide — a gas linked to ma­rine life on Earth.

But a Nasa-led re­analy­sis in 2025 found the sig­nal too weak to con­firm, and showed the gas can form with­out bi­ol­ogy.

The seven rocky worlds of TRAPPIST-1 re­main tan­ta­lis­ing, too: Nasa’s James Webb Space Telescope ruled out an Earth-like at­mos­phere on TRAPPIST-1d, while TRAPPIST-1e’s data stay frus­trat­ingly in­con­clu­sive.

V1.0 · Recurring · July 2026

A Letter From Our CTO, Raffi Krikorian “

In New Zealand’s far north, a Māori broad­caster trains speech mod­els for te reo — a lan­guage too small for any mar­ket — un­der a li­cense that keeps the data with its peo­ple. PwC, one of the largest ac­count­ing firms in the world, fine-tuned an open model on the lan­guage of fi­nance and runs it to­day for hun­dreds of clients, on its own hard­ware, with no per-to­ken me­ter run­ning. Researchers in Lausanne built an open med­ical model with the Red Cross, tuned to its hu­man­i­tar­ian guide­lines, and are prepar­ing clin­i­cal tri­als at home and in Tanzania. In East Africa, farm­ers di­ag­nose cas­sava dis­ease with a model that runs on the phone it­self, of­fline, in fields the cloud has never reached. In Switzerland, a pub­lic con­sor­tium trained a na­tional model on pub­lic su­per­com­put­ers and re­leased all of it: weights, data, train­ing code. None of them asked per­mis­sion, and none of them could have rented this. They own it — that is the whole idea.

We have been here be­fore. Mozilla ex­ists be­cause one com­pany tried to own the front door to the web, and an open com­mu­nity rose up to make sure it never could. Twenty-five years later, some­one is run­ning the same play. We bet on open the first time. Open won. Together, we can do it again.

Our be­lief is sim­ple: the path for­ward is com­pe­ti­tion and in­ter­op­er­abil­ity. We be­lieve in a world of many mod­els, stan­dard ways to plug them to­gether, and the free­dom to walk away from any ven­dor at any time. Open has a record here. It grew the pie and let more peo­ple own a slice of it.

Read what fol­lows as a map: where open AI is win­ning — some num­bers sur­prised even us — and where it is ex­posed. A case that hides its weak points is an ad­ver­tise­ment.”

Open weights closed the ca­pa­bil­ity gap while the price of in­tel­li­gence col­lapsed.

0%

Capability gap to the top closed mod­els — at par­ity on cod­ing, be­hind on rea­son­ing

0×

Fall in GPT-4-class in­fer­ence cost in 36 months: $20 → $0.40 per 1M to­kens

01The cur­rent state of open-source AI

Parity reached. The con­test is one layer up.

Open weights are no longer a com­pro­mise. They are where the work hap­pens: a ma­jor­ity of pro­duc­tion to­kens now route through them, and the five high­est-vol­ume mod­els on OpenRouter are all open. Closed mod­els still lead at the fron­tier, on rea­son­ing and mul­ti­modal­ity, but the fron­tier is not what most work­loads need. Commodity in­puts do not hold pric­ing power. Value moves up, to the agen­tic har­ness.

The ca­pa­bil­ity gap: 8.04% → 0.5% → 3.3%

Open-vs-closed gap on Chatbot Arena over 24 months. By August 2024, the gap had col­lapsed to 0.5%, and in February 2025 DeepSeek-R1 briefly matched the top US model. By March 2026 it had re­opened to 3.3% as closed rea­son­ing mod­els pulled ahead. But 3.3% is an av­er­age over a jagged fron­tier: open is at or near par­ity on cod­ing, in­struc­tion-fol­low­ing and gen­eral knowl­edge, while the gap con­cen­trates in rea­son­ing, long-con­text re­trieval and agen­tic tasks. The ques­tion is no longer whether open mod­els are good enough. It’s what you need for your work­load. Hover the points.

Source: Chatbot Arena, Jan 2024 — Mar 2026.

Inference fell 50× in 36 months

GPT-4-equivalent price per 1M to­kens — faster than dot­com-era band­width or PC-compute price curves. Log scale.

Sources: Stanford HAI AI Index 2025 (280× GPT-3.5-class drop over 18 months); Epoch AI (9 – 900× an­nual de­cay); Nov 2025 MIT study (5 – 10×/yr at the fron­tier, hard­ware-ad­justed).

Open weights win the to­kens

The share of to­kens routed on OpenRouter through open-weight mod­els grew from a neg­li­gi­ble base to a third by late 2025 to a ma­jor­ity by mid-2026.

Source: OpenRouter 100T-token study (Nov 2024–Nov 2025) and live leader­board; in­ter­me­di­ate points in­ter­po­lated. By re­quest count, closed US providers still lead — the open lead is a to­ken-vol­ume lead, con­cen­trated in cod­ing and agen­tic work­loads.

OpenRouter live leader­board — trail­ing month, to­kens routed

The five high­est-vol­ume mod­els are all open weights. Anthropic’s closed Claude mod­els are the next US-built en­trants.

Open weightsClosed

By mid-2026 the top nine mod­els route roughly 18T weekly to­kens for Chinese-built mod­els against ~5.5T for US-built ones — more than 3:1 (FT analy­sis). Where de­vel­op­ers route by cost, they route to open weights.

Open ships easy.Open de­ploys hard.

Data from the Mozilla / SlashData 2026 de­vel­oper sur­vey. Open mod­els lead in adop­tion: 79% of de­vel­op­ers adding AI func­tion­al­ity use them, against 71% for closed, and the two are largely com­ple­men­tary, with half of de­vel­op­ers us­ing both. But pro­duc­tion is where teams stall: only 51% of open-model teams reach pro­duc­tion ver­sus 63% for closed. The gap is op­er­a­tional tool­ing and trust, not model ca­pa­bil­ity.

Open mod­els lead in adop­tion, and mostly co­ex­ist with closed

Share of de­vel­op­ers adding AI func­tion­al­ity to their ap­pli­ca­tions who cur­rently use each model type, and how the two over­lap.

Open mod­els

79%

Closed mod­els

71%

How they com­bine

29%OS only

50%Both

21%CS only

Source: Mozilla / SlashData 2026 de­vel­oper sur­vey. Open and closed aren’t sub­sti­tutes for most teams: 50% run both, 29% open only, 21% closed only.

Where open adop­tion peaks, and where closed still edges it

Open-model adop­tion by re­gion. Greater China and East Asia lead at 89%; South America and Western Europe are the only two re­gions where closed adop­tion ex­ceeds open.

Same sur­vey, by de­vel­oper re­gion. In South America and Western Europe, and only there, closed-model adop­tion runs ahead of open.

Production rate by com­pany size

If the gap were about re­sources, scale would close it, and it does­n’t. Closed climbs 54% → 73% with scale. Open barely moves: 53% → 57%.

Closed mod­el­sOpen mod­els

Enterprises can buy their way through closed de­ploy­ment. Open de­ploy­ment waits on tool­ing no­body has fin­ished. Source: Mozilla / SlashData 2026 de­vel­oper sur­vey.

Why teams churn: chal­lenges with open mod­els

Δ = churned − still us­ing, in per­cent­age points. The biggest gaps (performance, in­te­gra­tion, main­te­nance) are op­er­a­tional, not ca­pa­bil­ity. Hover the bars.

Still us­ing openChurned away

Mozilla sur­vey, n=1,410. “What are the main chal­lenges you face when work­ing with open or open-source AI mod­els?”

The same chal­lenges, every­where: what blocks open by re­gion

Share of cur­rent and churned open-model de­vel­op­ers nam­ing each chal­lenge, by re­gion. Warmer cells mean more de­vel­op­ers blocked. The top rows are op­er­a­tional in every re­gion: in­fra­struc­ture cost, se­cu­rity and com­pli­ance, main­te­nance, de­ploy­ment com­plex­ity. South Asia leans hard­est on se­cu­rity and sup­port; only North America and Greater China have more than 15% re­port­ing no ma­jor chal­lenges.

Source: Mozilla / SlashData 2026 de­vel­oper sur­vey (MZCS1). n=1,410 cur­rent or churned open-model de­vel­op­ers; the Oceania col­umn (n=39) and Eastern Europe & CIS (n=98) fall be­low re­li­able thresh­olds.

02The open-source AI stack

The open stack scores high on ca­pa­bil­ity,low on op­er­a­tions.

Nine lay­ers and 48 com­po­nents of the stack scored across 10 cri­te­ria (1 – 5). Click a layer to open its com­po­nents: each car­ries its own cri­te­rion scores, ma­tu­rity grade, open-vs-closed par­ity ver­dict, and sur­faces some of its most-starred open-source pro­jects.

Hover any cell for de­tail.

StrongViable, but frag­ment­edEarly stage

Strong (≥4.0) 3.5 – 3.9 3.0 – 3.4 2.5 – 2.9 Weak (<2.5) the op­er­a­tional gap = stan­dard­iza­tion + en­ter­prise readi­ness

Cells are scores per ma­tu­rity cri­te­rion (1 – 5), or­dered strongest to weak­est left to right; layer rows are the means of their com­po­nents. The two cold­est columns, stan­dard­iza­tion and en­ter­prise readi­ness, re­peat down every layer and every com­po­nent: that re­peat­ing cold edge is the op­er­a­tional gap. Source: Mozilla stack map, June 2026 (48 com­po­nents, 1,361 pro­jects).

03Who’s bet­ting on it

Open source is a busi­ness model.

Open-weight AI is a com­mer­cial mar­ket at multi-hun­dred-bil­lion-dol­lar scale, built by funded com­pa­nies and run in pro­duc­tion by global en­ter­prises. Databricks crossed a $5.4B run-rate; Mistral scaled 20× to ~$400M ARR in twelve months; DeepSeek reached ~$220M ARR and re­cently raised $7.4B at a val­u­a­tion over $50B. Five rev­enue mod­els are proven at scale: hosted in­fer­ence, en­ter­prise plat­forms, on-prem li­cens­ing, fine-tun­ing ser­vices, and har­ness tool­ing.

The ven­ture-funded open-source ecosys­tem: to­tal dis­closed fund­ing, USD M

Bars grow as you scroll. Color by re­gion of the com­pany.

North AmericaChinaEurope & rest of world

Selected com­pa­nies; Zhipu AI and MiniMax went pub­lic (HK IPO 2026) with undis­closed to­tals. Corporate strate­gics (Nvidia, Salesforce, AMD, Google, IBM, ASML, Tencent, CATL, Schwarz Group) back the same ecosys­tem across model, in­fer­ence, and tool­ing lay­ers.

Financial ma­tu­rity of the open ecosys­tem

Funding, val­u­a­tion and rev­enue trac­tion for the com­pa­nies car­ry­ing the open stack. The ecosys­tem has moved from grants to ven­ture scale to pub­lic mar­kets.

Five rev­enue mod­els are proven at scale: hosted in­fer­ence, en­ter­prise plat­forms, on-prem li­cens­ing, fine-tun­ing ser­vices, and har­ness tool­ing. “—” = not pub­licly dis­closed.

The me­tered model breaks at scale

Closed fron­tier mod­els are sold by the to­ken — and at pro­duc­tion scale the me­ter be­comes the prob­lem.

A fifth of the us­age, 4% of the rev­enue

On OpenRouter (May–Sep 2025), closed mod­els held ~80% of us­age and ~96% of rev­enue. Price dri­ves it: at ~90% par­ity, closed costs ~6× more per call.

~$24.8B

in un­re­al­ized an­nual sav­ings — the Nagle–Yue study for the Linux Foundation’s es­ti­mate of the open-vs-closed price asym­me­try, at ~6× the cost per call for com­pa­ra­ble ca­pa­bil­ity

Where de­vel­op­ers route by cost, they route to open weights.

04Why it’s hap­pen­ing every­where

Open is­n’t a ven­dor choice.It’s a sov­er­eignty choice.

More than 70 na­tional AI strate­gies are live. The strate­gic ques­tion has shifted from whether to have a na­tional AI pol­icy to which layer of the stack a coun­try can own.

Click a marker or a coun­try be­low.

The case for open is op­tion­al­ity

Optionality stopped be­ing ab­stract in June 2026, and it stopped be­ing a pro­cure­ment ques­tion. Three days af­ter Claude Fable 5 went on sale, a sin­gle gov­ern­men­t’s ex­port or­der forced Anthropic to cut ac­cess for every for­eign na­tional on earth. No other cap­i­tal was con­sulted. None could have been. Selective com­pli­ance was im­pos­si­ble, so the mod­els went dark for every­one at 5:21 p.m. on a Friday. Anyone who had built on that model in­her­ited a shut­down they had no warn­ing of and no part in. A provider can switch off a model. Nobody can switch off a copy al­ready run­ning on a ma­chine you hold, and that holds whether the ma­chine is a star­tup’s server or a na­tional su­per­com­puter. For a com­pany, weights on disk are a hedge. For a state, they are the dif­fer­ence be­tween a pol­icy and a per­mis­sion.

The strate­gic case for open is the abil­ity to leave, and the cloud era proved the cost of its ab­sence:

$90 – 120kto move one petabyte out of AWS S3

80%of en­ter­prises now repa­tri­at­ing work­loads

$3.2M → <$1M37signals’ cloud bill af­ter leav­ing

2.5×what GEICO’s cloud costs ran over plan

Closed model APIs re­pro­duce the same trap: build on a pro­pri­etary end­point and you in­herit the ven­dor’s pric­ing changes with no clean exit. Open weights are exit rights.

The largest source of open weights is China. By de­sign.

Cumulative Hugging Face down­loads, March 2026:

In February 2026 Qwen out-down­loaded the next eight or­ga­ni­za­tions com­bined. On OpenRouter, Chinese open-weight mod­els rose from un­der 2% of to­kens in late 2024 to more than 45% of weekly traf­fic by April 2026, and about 61% among the ten most-used mod­els. DeepSeek re­ports 26,000+ en­ter­prise ac­counts; 58% of new AI star­tups in 2025 in­cluded it in their stack, even as at least eight ju­ris­dic­tions re­stricted the hosted ser­vice. The res­o­lu­tion is ar­chi­tec­tural: en­ter­prises ban the hosted app and adopt the weights any­way, self-hosted or via Western end­points.

This is in­ten­tional pol­icy. The State Council’s “AI Plus” Initiative (Aug 2025) and the na­tional Five-Year Plan (Mar 2026) cod­ify open-source pro­lif­er­a­tion as a core di­rec­tive, and re­leas­ing pub­lic weights dou­bles as a macro hedge against semi­con­duc­tor ex­port con­trols, of­fload­ing global in­fer­ence onto end users’ lo­cal hard­ware. Across the Global South the draw is di­ver­si­fi­ca­tion away from US tech­nol­ogy mo­nop­o­lies; else­where it is purely fi­nan­cial. Even Microsoft is ex­plor­ing a se­cured, Azure-hosted DeepSeek V4 for its heav­i­est Copilot work­load.

Marker size ≈ scale of com­mit­ted pub­lic/​strate­gic cap­i­tal · Equirectangular pro­jec­tion

Source: Open Source AI ju­ris­dic­tions dataset, July 2026. Marker size scales with com­mit­ted pub­lic and strate­gic cap­i­tal.

05The har­ness is the new fron­tier

The agen­tic har­ness is an­other user agent.

The browser was the user agent of the open web: code on the user’s side, ne­go­ti­at­ing with servers on their be­half. That role is be­ing recre­ated one layer up. Above the model now sits the agen­tic har­ness — the or­ches­tra­tion loop, tools, mem­ory, sand­boxes, and per­mis­sion model. It is where pro­duc­tion dif­fi­culty con­cen­trates, and where the open-vs-closed, owner-vs-renter con­test restarts.

The user · other agents · the world­hu­mans · sys­tems · data · money

For help please visit help.ft.com. We apol­o­gise for any in­con­ve­nience.

The fol­low­ing in­for­ma­tion can help our sup­port team to re­solve this is­sue.

16th July 2026

Chinese AI lab Moonshot AI an­nounced Kimi K3 this morn­ing, de­scrib­ing it as their “most ca­pa­ble model to date, with 2.8 tril­lion pa­ra­me­ters”. It’s cur­rently avail­able via their web­site and API, but an open weight re­lease is promised “by July 27, 2026”.

Moonshot are call­ing this the first “open 3T-class model” (I guess they’re round­ing 2.8 tril­lion up to 3 tril­lion), tak­ing the crown from DeepSeek’s 1.6T v4 Pro. Their self-re­ported bench­marks have K3 mostly beat­ing Claude Opus 4.8 max and GPT-5.5 high, while los­ing out to Claude Fable 5 and GPT-5.6 Sol.

A few high­lights from the Artificial Analysis re­port on the model:

“On our pri­vate long-hori­zon knowl­edge work eval­u­a­tion, Kimi K3 reaches an over­all Elo of 1547, +732 points from Kimi K2.6 and be­hind only Claude Fable 5.”

“Cost per task ($0.94) is sim­i­lar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers”

“Kimi K3’s to­ken us­age on the Artificial Analysis Intelligence Index de­creased sig­nif­i­cantly, us­ing 21% fewer out­put to­kens than K2.6.”

The model is also now the lead­ing model on Arena.ai’s Frontend Code arena, sur­pass­ing even Claude Fable 5.

The new model is no­table for the pric­ing: $3/million in­put to­kens and $15/million out­put to­kens, putting it at the same level as Anthropic’s Claude Sonnet se­ries and mak­ing it the most ex­pen­sive model re­leased by a Chinese AI lab to date. This is a sig­nif­i­cant in­crease on their ear­lier mod­els such as Kimi K2.6 at $0.95/$4. 2.8 tril­lion pa­ra­me­ters is also more than twice the size of that 1T model.

But how does it pel­i­can?

I used OpenRouter (to avoid sign­ing up for a Moonshot API key) with the llm-open­router plu­gin to gen­er­ate an SVG of a pel­i­can rid­ing a bi­cy­cle:

llm -m open­router/​moon­shotai/​kimi-k3 ‘Generate an SVG of a pel­i­can rid­ing a bi­cy­cle’

Here’s the tran­script. It looks like this:

That pel­i­can took 95 in­put to­kens and 16,658 out­put to­kens (13,241 were rea­son­ing to­kens), for a to­tal cost of 25 cents!

Since K3 ac­cepts im­age in­put I ran it against that ren­dered SVG above (with my alt text prompt) and got back (for 0.6 cents):

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

What can we learn from the pel­i­can?

My Generate an SVG of a pel­i­can rid­ing a bi­cy­cle test is 21 months old now. It was never a par­tic­u­larly great bench­mark. It started out as a joke on how ab­surdly dif­fi­cult it is to com­pare these mod­els, but then for the first year it turned out to have a sur­pris­ing cor­re­la­tion to how good the mod­els ac­tu­ally were.

That con­nec­tion has been mostly sev­ered now. The GPT-5.6 and Claude Fable 5 pel­i­cans are out­classed by GLM-5.2, and much as I love GLM I don’t think that’s a Fable-class model.

(I’m still not con­vinced that labs are train­ing for the bench­mark—if they were, I’d ex­pect much bet­ter re­sults. There’s a chance that Gemini has op­ti­mized for any com­bi­na­tion of an an­i­mal on a ve­hi­cle though!)

The biggest lim­i­ta­tion of the pel­i­can is that it does­n’t touch at all on the thing that mat­ters most for to­day’s model: agen­tic tool call­ing and the abil­ity to op­er­ate tools re­li­ably as con­ver­sa­tions grow in length.

So don’t go us­ing pel­i­cans to com­pare mod­els!

All of that said, I still get a de­cent amount of value out of run­ning the bench­mark my­self.

Firstly, it’s a forc­ing func­tion for ac­tu­ally try­ing the model. If I show you a pel­i­can, that means I’ve man­aged to run a prompt through it. If the model has an of­fi­cial API I’ll use that, if it’s open weight (and small enough to fit a 128GB M5 MacBook Pro) I’ll try run­ning it on my own ma­chine, usu­ally via llama.cpp or LM Studio or Ollama. I’ll fre­quently use OpenRouter since that usu­ally pro­vides a proxy to an of­fi­cial API with­out me need­ing a new API key.

Most of my pel­i­cans are gen­er­ated us­ing my LLM CLI tool, which helps en­cour­age me to en­sure the lat­est mod­els are sup­ported by that (via one of its plu­g­ins).

More im­por­tantly though, even the act of a sin­gle prompt to “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” can re­veal in­ter­est­ing model char­ac­ter­is­tics.

Consider the re­sult for Kimi K3 to­day. Running those sim­ple prompts helped em­pha­size sev­eral points about the model.

It only has one rea­son­ing ef­fort right now, “max”—and it shows. The model con­sumed 13,241 rea­son­ing to­kens to out­put 3,417 to­kens of re­sponse. This is ex­pen­sive—the pel­i­can cost 25 cents!

How does the prompt “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” add up to 95 in­put to­kens? OpenAI’s to­k­enizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 to­kens, sug­gest­ing there may be an 85 to­ken hid­den sys­tem prompt. It re­fused to leak it though.

Vision works well: the alt text it gen­er­ated is very good.

K3 cur­rently only has one think­ing ef­fort level, but I’ve been de­riv­ing quite a bit of value re­cently from run­ning the same pel­i­can prompt through dif­fer­ent ef­fort lev­els to get a quick idea for what im­pact those have. Here’s my ma­trix for the GPT-5.6 model fam­ily, for ex­am­ple.

Really though the main things I gain from the pel­i­can test are:

It’s a “hello world” ex­er­cise for prompt­ing a model

A rough cost and rea­son­ing es­ti­mate for a sim­ple task

Confirmation that the model can out­put valid SVG and has a ba­sic idea of geom­e­try and spa­tial aware­ness. This is a much big­ger deal for the smaller mod­els that run on my lap­top.

It’s still in­ter­est­ing to com­pare pel­i­cans be­tween re­leases in the same model fam­ily. K3’s pel­i­can is a no­table im­prove­ment from Kimi 2.5.

It’s some­thing I can share that demon­strates I’ve tried it. Plus a com­ment with a pel­i­can in it is kind of a tra­di­tion on Hacker News at this point, any time I’m late I get com­ments ask­ing where it is!

Loading met­rics

Loading met­rics

Open Access

Peer-reviewed

Research Article

Emina Aličković,

Johannes Zaar,

Alejandro López Valdés  ,

Giovanni M. Di Liberto

Competing speech streams are si­mul­ta­ne­ously rep­re­sented in the hu­man cor­tex dur­ing at­ten­tion switch­ing

Sara Carta,

Emina Aličković,

Johannes Zaar,

Alejandro López Valdés,

Giovanni M. Di Liberto

x

Published: July 16, 2026

https://​doi.org/​10.1371/​jour­nal.pbio.3003876

Figures

Abstract

Successful speech com­mu­ni­ca­tion in multi-talker sce­nar­ios re­quires a skill­ful com­bi­na­tion of sus­tained at­ten­tion and rapid at­ten­tion switch­ing. While the neu­ro­phys­i­ol­ogy lit­er­a­ture of­fers de­tailed in­sights into the neural un­der­pin­nings of sus­tained at­ten­tion, there re­mains con­sid­er­able un­cer­tainty on how at­ten­tion switch­ing takes place. In this study, us­ing EEG record­ings from nor­mal-hear­ing adults in an im­mer­sive multi-talker en­vi­ron­ment, we mea­sured the neural en­cod­ing of two com­pet­ing speech streams amid back­ground bab­ble. Participants were cued to switch at­ten­tion be­tween streams every 15 – 30 s. Neural track­ing was as­sessed via Temporal Response Functions (TRF), con­firm­ing re­li­able de­cod­ing of at­ten­tional fo­cus. Our re­sults in­di­cate asym­met­ric dis­en­gage­ment and en­gage­ment processes dur­ing at­ten­tion switches, where the neural track­ing of the new tar­get stream emerges be­fore dis­en­gag­ing from the pre­vi­ous tar­get, re­veal­ing a tran­sient si­mul­ta­ne­ous en­cod­ing of two speech streams. That tran­si­tion was closely mir­rored by a re­duc­tion in EEG al­pha power, in­form­ing on the cog­ni­tive ef­fort dur­ing dif­fer­ent phases of the at­ten­tion switch. We then iso­lated cor­ti­cal ac­tiv­ity re­flect­ing lex­i­cal pre­dic­tion mech­a­nisms to de­ter­mine how lex­i­cal con­text is up­dated af­ter an at­ten­tion switch, com­par­ing four con­text-ac­cu­mu­la­tion strate­gies that were con­structed us­ing Large Language Models. Our find­ings elu­ci­date both the tem­po­ral and con­tex­tual mech­a­nisms un­der­ly­ing au­di­tory at­ten­tion shifts, point­ing to the pos­si­bil­ity that lis­ten­ers carry out a re­set in lex­i­cal con­text af­ter switch­ing at­ten­tion. By fo­cus­ing on dy­namic at­ten­tional re­al­lo­ca­tion, this study of­fers in­sights into the brain’s ca­pac­ity for flex­i­ble speech pro­cess­ing in com­plex lis­ten­ing en­vi­ron­ments.

Citation: Carta S, Aličković E, Zaar J, López Valdés A, Di Liberto GM (2026) Competing speech streams are si­mul­ta­ne­ously rep­re­sented in the hu­man cor­tex dur­ing at­ten­tion switch­ing. PLoS Biol 24(7): e3003876.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876

Academic Editor: Manuel S. Malmierca, Universidad de Salamanca, SPAIN

Received: July 3, 2025; Accepted: June 12, 2026; Published: July 16, 2026

Copyright: © 2026 Carta et al. This is an open ac­cess ar­ti­cle dis­trib­uted un­der the terms of the Creative Commons Attribution License, which per­mits un­re­stricted use, dis­tri­b­u­tion, and re­pro­duc­tion in any medium, pro­vided the orig­i­nal au­thor and source are cred­ited.

Data Availability: All data sup­port­ing the find­ings re­ported in this man­u­script are freely ac­ces­si­ble with­out re­stric­tion. The EEG pre-processed dataset, the re­sult­ing analy­sis files, and the analy­sis code are pub­licly avail­able on the open repos­i­tory Zenodo (https://​zen­odo.org/​records/​20569817). The EEG record­ings are pro­vided fol­low­ing the Continuous-event Neural Data (CND) for­mat stan­dard. The as­so­ci­ated speech stim­uli can also be found in the same repos­i­tory, within the STIMULI folder.

Funding: S.C., A.L.V., and G.D.L. were sup­ported by the William Demant Fonden (https://​www.williamde­mant­fonden.dk/), un­der grants 21 – 0628 and 22 – 0552, and by Taighde Éireann — Research Ireland (https://​www.re­searchire­land.ie/) un­der grant No. 18/CRT/6223. G.D.L. ad­di­tion­ally con­ducted this re­search with the fi­nan­cial sup­port of Research Ireland at ADAPT, the Research Ireland Centre for AI-Driven Digital Content Technology (https://​www.adapt­cen­tre.ie/) at Trinity College Dublin [grant 13/RC/2106_P2]. The fun­ders had no role in study de­sign, data col­lec­tion and analy­sis, de­ci­sion to pub­lish, or prepa­ra­tion of the man­u­script.

Competing in­ter­ests: The au­thors have de­clared that no com­pet­ing in­ter­ests ex­ist.

Abbreviations: EEG, elec­troen­cephalog­ra­phy; EOG, elec­tro-ocu­log­ra­phy; EMG, elec­tro-myo­g­ra­phy; ERSP, event-re­lated spec­tral per­tur­ba­tion; iEEG, in­tra-cra­nial elec­troen­cephalog­ra­phy; FDR, false dis­cov­ery rate; fMRI, func­tional mag­netic res­o­nance imag­ing; ICA, Independent Component Analysis; IQR, in­terquar­tile range; LLM, large lan­guage model; MEG, mag­ne­toen­cephalog­ra­phy; PSD, power spec­tral den­sity; RMS, root-mean-squared; SE, stan­dard er­ror; SEM, stan­dard er­ror of the mean; SNR, sig­nal-to-noise ra­tio; SPL, sound pres­sure level; TRF, Temporal Response Functions

Introduction

To un­der­stand speech in multi-talker en­vi­ron­ments, lis­ten­ers sin­gle out the tar­get speaker from com­pet­ing sound streams [1 – 3]. The neu­ro­phys­i­ol­ogy of this se­lec­tive at­ten­tion process has been widely stud­ied with sim­u­lated cock­tail-party sce­nar­ios [4,5], shed­ding light on how our brains seg­re­gate a tar­get stream from com­pet­ing speech streams, and en­abling the trans­for­ma­tion of the tar­get speech into lin­guis­tic mean­ing. While the ex­tent to which masker speech streams are processed re­mains highly de­bated [6 – 8], there is no doubt that there are con­sid­er­able dif­fer­ences be­tween the pro­cess­ing of tar­get and masker speech, which have been mea­sured with var­i­ous tech­nolo­gies, such as non-in­va­sive elec­troen­cephalog­ra­phy (EEG) [1,9], in­tra-cra­nial elec­troen­cephalog­ra­phy (iEEG) [10], mag­ne­toen­cephalog­ra­phy (MEG) [3,11] and func­tional mag­netic res­o­nance imag­ing (fMRI) [12,13]. That work could pin­point pre­cise loci in the au­di­tory cor­ti­cal ar­eas where that seg­re­ga­tion emerges [14] as well as mea­sur­ing the sub­stan­tial (but not to­tal) sup­pres­sion of lin­guis­tic pro­cess­ing for the masker speech [1,15 – 17]. However, neu­ro­phys­i­ol­ogy lit­er­a­ture in this field has al­most en­tirely fo­cused on sus­tained at­ten­tion tasks [2,10], leav­ing con­sid­er­able un­cer­tainty on the neural un­der­pin­nings of at­ten­tion switch­ing.

Dynamic switch­ing par­a­digms have been widely used in the do­main of cog­ni­tive con­trol stud­ies to probe for cog­ni­tive flex­i­bil­ity and cog­ni­tive sta­bil­ity [18]. In those ex­per­i­ments, par­tic­i­pants are of­ten re­quired to flex­i­bly adapt their be­hav­ioral re­sponse de­pend­ing on new in­struc­tions, ini­ti­at­ing a task-switch [19 – 21]. For ex­am­ple, given a sin­gle digit, they are re­quired to clas­sify it ei­ther based on par­ity, i.e., whether it is even or odd, or based on rel­a­tive mag­ni­tude, i.e., whether the digit is greater than or less than 5 [22]. In these par­a­digms, the switch-cost is the in­crease in re­ac­tion time or er­ror rate when switch­ing from one task to the other. Similar be­hav­ioral par­a­digms have also in­volved sim­ple speech stim­uli in multi-talker set­tings [23 – 25]. However, the main in­ter­est of those tightly con­trolled ex­per­i­ments was to model the process of tar­get speech se­lec­tion as one par­tic­u­lar in­stance of a task-switch­ing prob­lem, i.e., tar­get stream se­lec­tion could ei­ther de­pend on spa­tial lo­ca­tion or voice iden­tity [23], rather than fo­cus­ing on the dy­namic as­pect of at­ten­tion re-al­lo­ca­tion per se in nat­u­ral­is­tic multi-talker sce­nar­ios. As such, very lit­tle is known on how a flex­i­ble re­ori­ent­ing of at­ten­tion might im­pact speech pro­cess­ing of con­tin­u­ous com­pet­ing streams.

In re­cent speech neu­ro­phys­i­ol­ogy re­search, ex­per­i­men­tal par­a­digms have started to in­clude switches of at­ten­tion as a tool to­wards tai­lored EEG/MEG method­olog­i­cal ad­vances in the do­main of at­ten­tion de­cod­ing [26,27], or to in­ves­ti­gate how sus­tained speech at­ten­tion un­folds for mov­ing au­di­tory ob­jects [28]. However, to the best of our knowl­edge, only one pre­vi­ous study has specif­i­cally fo­cused on the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing in multi-talker sce­nar­ios, re­lat­ing the neural en­cod­ing of speech dur­ing at­ten­tional re-ori­ent­ing with EEG al­pha ac­tiv­ity and pupil di­la­tion dy­nam­ics [29]. Those find­ings proved that the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing can be stud­ied non-in­va­sively. Building on that work, our study sheds light on the ex­act neural dy­nam­ics sup­port­ing the steer­ing of at­ten­tion be­tween two com­pet­ing speech streams, dis­en­gag­ing from the pre­vi­ous tar­get stream while en­gag­ing to the new one.

In this study, we mea­sure the neural en­cod­ing of speech us­ing a range of en­cod­ing win­dow lengths, as lis­ten­ers steer their at­ten­tion from one speaker to an­other. We test whether en­gage­ment with a new speech stream be­gins be­fore dis­en­gage­ment from the pre­vi­ous tar­get is com­plete, re­sult­ing in a brief pe­riod of si­mul­ta­ne­ous track­ing of both streams. Such an asym­me­try in the dis­en­gage­ment-en­gage­ment processes, even if tran­sient, could sup­port the abil­ity to ex­plore al­ter­na­tive au­di­tory streams while main­tain­ing at­ten­tion to a given stream [30].

The neural en­cod­ing of speech was mea­sured from nor­mal-hear­ing adult par­tic­i­pants us­ing EEG dur­ing an im­mer­sive multi-talker lis­ten­ing task. Participants were ex­posed to two com­pet­ing speech streams from TED talks, pre­sented via two front-fac­ing loud­speak­ers, while back­ground noise from a 16-talker speech bab­ble played from rear loud­speak­ers (Fig 1A). An on-screen ar­row cued par­tic­i­pants to at­tend to one of the two speech streams and to shift their at­ten­tion rapidly when­ever the ar­row changed di­rec­tion, ap­prox­i­mately every 10 – 30 s (Fig 1B). Neural track­ing of tar­get and masker speech was quan­ti­fied us­ing the Temporal Response Function (TRF), de­scrib­ing the lin­ear re­la­tion­ship be­tween each speech stream and the neural re­sponses. As an ini­tial val­i­da­tion, we con­firmed that the at­tended stream could be re­li­ably de­coded from the EEG, con­sis­tent with the ex­ten­sive lit­er­a­ture on sus­tained at­ten­tion [9,10,31]. This con­firms that the EEG re­sponses in this ex­per­i­ment re­flects dif­fer­en­tial en­cod­ing of tar­get ver­sus masker speech (Fig 1C).

Fig 1. Experiment overview and val­i­da­tion.

(A) Participants were pre­sented with speech from two loud­speak­ers placed in front of them with 60° of sep­a­ra­tion (30° left and 30° right), and with con­cur­rent 16-talker back­ground noise (B1–B4). In each trial, the screen pre­sented an ar­row point­ing to the tar­get speech stream. Participants were in­structed to switch at­ten­tion as soon as the vi­sual cue changes di­rec­tion. (B) Schematic di­a­gram of one ex­per­i­men­tal trial. The black area rep­re­sents blocks of at­ten­tion ei­ther to the left (L) or right (R) front streams. The red ar­rows in­di­cate the in­stants where the at­ten­tion cue switches side (six times per trial). Note that block du­ra­tion was ran­dom­ized and al­ways be­tween 15 and 30 s, with tri­als last­ing 3 min. (C) EEG data val­i­da­tion was car­ried out by run­ning an at­ten­tion de­cod­ing analy­sis. Progressively longer de­cod­ing win­dows were con­sid­ered (larger win­dows use more data, typ­i­cally lead­ing to more ac­cu­rate de­cod­ing scores). Binary clas­si­fi­ca­tion scores are re­ported ar­bi­trat­ing be­tween the tar­get and masker streams. The dashed line in­di­cates the 95th per­centile of a ran­dom dis­tri­b­u­tion cal­cu­lated by ran­dom­iz­ing the clas­si­fi­ca­tion la­bels. Statistically sig­nif­i­cant at­ten­tion de­cod­ing clas­si­fi­ca­tion scores were mea­sured for all the de­cod­ing win­dows con­sid­ered, with nu­mer­i­cal re­sults com­pa­ra­ble with pre­vi­ous stud­ies on se­lec­tive at­ten­tion [31,34,35]. Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g001

We next ad­dressed two fun­da­men­tal ques­tions about the neural mech­a­nisms un­der­ly­ing at­ten­tion switch­ing in nat­u­ral­is­tic lis­ten­ing. First, we asked whether the processes of en­gag­ing with a new speech stream and dis­en­gag­ing from a pre­vi­ous one un­fold sym­met­ri­cally (Figs 2 and 3). To test this, we fit en­cod­ing TRF mod­els to EEG data, mea­sur­ing the neural track­ing of the two com­pet­ing speech streams over time. This al­lowed us to char­ac­ter­ize the av­er­age en­cod­ing dy­nam­ics sur­round­ing at­ten­tion switches, com­par­ing dis­en­gage­ment and en­gage­ment processes. The sec­ond ob­jec­tive was to un­der­stand how our brains up­date and use lex­i­cal con­text when switch­ing at­ten­tion (Fig 4). Building on pre­vi­ous work show­ing that speech com­pre­hen­sion is sup­ported by con­tex­tual pre­dic­tions [32,33], we for­mu­lated four com­pet­ing hy­pothe­ses re­flect­ing dif­fer­ent as­sump­tions about how lin­guis­tic con­text is pre­served, re­set, or se­lec­tively up­dated across an at­ten­tion switch. Using a state-of-the-art large lan­guage model (LLM), we de­rived quan­ti­ta­tive pre­dic­tions for each hy­poth­e­sis, re­sult­ing in four re­gres­sors for lex­i­cal sur­prisal and en­tropy, sep­a­rately, dif­fer­ing in their sen­si­tiv­ity to prior con­text and to the oc­cur­rence of the switch. Encoding TRF mod­els were then fit for each hy­poth­e­sis, al­low­ing us to com­pare al­ter­na­tive con­text-ac­cu­mu­la­tion strate­gies and iden­tify the model most con­sis­tent with the ob­served neural re­sponses. This study pro­vides sub­stan­tial new in­sights into the tem­po­ral un­fold­ing and con­tex­tual mech­a­nisms guid­ing at­ten­tion switch­ing, en­com­pass­ing both low and high lev­els of speech ab­strac­tion.

Fig 2. The at­ten­tion-switch­ing cue prompts a ro­bust dis­en­gage­ment from Speaker 1 and en­gage­ment to Speaker 2, and it is fol­lowed by a sig­nif­i­cant de­crease in the EEG al­pha ERSP.

Disengagement has longer tem­po­ral dy­nam­ics com­pared to en­gage­ment. (A) Left: Speech track­ing en­cod­ing for an at­ten­tion switch from Speaker 1 and 2. The tra­jec­tory in the panel rep­re­sents our null hy­poth­e­sis, where the dis­en­gage­ment and en­gage­ment processes progress in a sym­met­ric man­ner af­ter the switch-cue (vertical gray line). Right: Results for the neural track­ing of Speaker 1 and Speaker 2 across the switch­ing cue. EEG pre­dic­tion cor­re­la­tions (average across all chan­nels) ob­tained from a 4-s slid­ing-win­dow TRF model in­clud­ing Envelope (Env), Word Onset (WO) and Word Surprisal (WS) fea­tures. Coloured hor­i­zon­tal bars at the bot­tom of the plot in­di­cate the at­ten­tion in­struc­tion around the at­ten­tion switch­ing cue. The turquoise dot in­di­cates the en­cod­ing switch of EEG pre­dic­tion cor­re­la­tions based on Spk1- and Spk2- speech fea­tures. The piece­wise lin­ear model fit for dis­en­gage­ment and en­gage­ment is over­layed on the EEG pre­dic­tion cor­re­la­tion val­ues. Please note that the bro­ken-line-fit in this plot was per­formed on the grand-av­er­age cor­ti­cal track­ing curves here for il­lus­tra­tive pur­poses. Please find the es­ti­mates at the sin­gle-par­tic­i­pant level in Panel C. Hexagram shapes in­di­cate the start of the dis­en­gage­ment (blue) and en­gage­ment (yellow) processes, while di­a­monds rep­re­sent the end of the tran­si­tions. (B) Left: Diagram of ex­pected re­sults for al­pha-band ERSP (event-related spec­tral per­tur­ba­tion) across the switch­ing cue. Right: ERSP of the al­pha band (8 – 12 Hz) around the switch­ing cue (average of all chan­nels), com­puted with a 4-s slid­ing win­dow, as above. Scalp topogra­phies at se­lected time points re­veal a pat­tern of pos­te­rior neg­a­tiv­ity, which drops sig­nif­i­cantly fol­low­ing the in­struc­tion to switch (thick black lines in­di­cate a sta­tis­ti­cally sig­nif­i­cant change com­pared to pre-switch base­line). The red dot rep­re­sents the av­er­age of ERSP min­ima across par­tic­i­pants. The shaded area rep­re­sents the stan­dard er­ror of the mean (SEM) across par­tic­i­pants. (C) Left: Comparison of en­cod­ing switch of EEG pre­dic­tion cor­re­la­tions (turquoise bar) and al­pha ERSP min­i­mum (red bar) for a 4-s slid­ing win­dow. The al­pha ERSP reaches its min­i­mum sig­nif­i­cantly af­ter the Spk1-Spk2 en­cod­ing switch point. Right: Comparison of tem­po­ral dy­nam­ics for start and end points of dis­en­gage­ment and en­gage­ment processes, with start/​end tran­si­tion points es­ti­mated at the sin­gle-par­tic­i­pant level. Stars in­di­cate sig­nif­i­cant sta­tis­ti­cal ef­fects (paired sam­ple t-tests; *p ≤ 0.05; **p ≤ 0.01; **p ≤ 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g002

Fig 3. Comparing the start and end tran­si­tion points for the dis­en­gage­ment and en­gage­ment processes af­ter the at­ten­tions switch­ing cue.

The process of en­gag­ing to a new speaker be­gins and ends sig­nif­i­cantly ear­lier than dis­en­gag­ing from the pre­vi­ously at­tended speaker. (A, B) Start and end points of the tran­si­tion for the dis­en­gage­ment (blue) and en­gage­ment (yellow) processes over five TRF slid­ing win­dow lengths. Error bars rep­re­sent SEM across par­tic­i­pants. Stars in­di­cate sig­nif­i­cant ef­fects of process type (two-way re­peated mea­sures ANOVA; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g003

Fig 4. Investigating lex­i­cal pre­dic­tion mech­a­nisms dur­ing at­ten­tion switch­ing.

(A) Layout of the four con­text mod­els. Blocks coloured in black il­lus­trate sus­tained at­ten­tion ei­ther to the Left or Right stream, while or­ange ar­rows in­di­cate at­ten­tion switch­ing cues. The thick red ar­row in­di­cates the con­text used to guide word pre­dic­tions for the cur­rent block (B7, high­lighted in or­ange). (B) Average lex­i­cal en­tropy at words pre­ced­ing and fol­low­ing the at­ten­tion switch cue. Note that no value for en­tropy is dis­played in the Reset model for the first word af­ter the switch, due to the con­text be­ing fully re­in­stated. (C) EEG pre­dic­tion cor­re­la­tions for the four mul­ti­vari­ate TRF mod­els, only dif­fer­ing in their en­tropy fea­ture. Coloured dots in­di­cate the av­er­age across all elec­trodes and par­tic­i­pants. The gray area at the bot­tom rep­re­sents the av­er­age en­cod­ing ac­cu­racy of a mul­ti­vari­ate TRF with­out any se­man­tic in­for­ma­tion (Envelope + Word Onset). Stars rep­re­sent sta­tis­ti­cally sig­nif­i­cantly greater EEG pre­dic­tion cor­re­la­tions for the Reset model com­pared to the other mod­els (Significance lev­els: *p < 0.05, **p < 0.01, ***p < 0.001). Topographical pat­terns il­lus­trate the gain due to se­man­tic in­for­ma­tion (compared to the Envelope + Word Onset TRF) for the four mod­els. (D) (Left) TRF weights for the en­tropy fea­ture at time-lags be­tween −100 and 600 ms rel­a­tive to stim­u­lus on­set. Transparent shaded ar­eas rep­re­sent the stan­dard er­ror of the mean (SEM) across par­tic­i­pants. The hor­i­zon­tal black line in­di­cates the time win­dow em­ployed to com­pute the av­er­age TRF-N400 am­pli­tude. (Right) Boxplots rep­re­sent­ing the dis­tri­b­u­tion of the TRF-N400 am­pli­tude across par­tic­i­pants for the four con­text mod­els. The cen­tral line within each box rep­re­sents the me­dian, while the edges of the box in­di­cate the in­terquar­tile range (IQR). Whiskers ex­tend to the most ex­treme data points within 1.5 times the IQR from the quar­tiles. Outliers are plot­ted as in­di­vid­ual points be­yond the whiskers. Stars in­di­cate sta­tis­ti­cally sig­nif­i­cant dif­fer­ences (Significance lev­els: *p < 0.05, **p < 0.01, ***p < 0.001). Data sup­port­ing this fig­ure is avail­able at: https://​zen­odo.org/​records/​20569817.

https://​doi.org/​10.1371/​jour­nal.pbio.3003876.g004

Results

Behavioral per­for­mance

Following each trial, par­tic­i­pants were first pre­sented with a four-al­ter­na­tive forced-choice ques­tion about the con­tent of the at­tended speech stream to con­firm task en­gage­ment. Behavioral per­for­mance re­vealed that they were able to suc­cess­fully re­ply to con­tent-re­lated ques­tions, with an av­er­age ac­cu­racy of 86.3% (SEM 2.6%). Participants were also re­quired to in­di­cate their pref­er­ence be­tween left and right streams, which was found to be over­all bal­anced, with the left stream se­lected in 49.79% of the tri­als, on av­er­age (SEM 1.7%). Finally, per­ceived dif­fi­culty of the at­ten­tion switch for every trial was mea­sured by ask­ing par­tic­i­pants to rate it on a scale from 1 (very easy) to 5 (very hard). The av­er­age dif­fi­culty of the switch was judged to be 3.1 out of 5, with a SEM across par­tic­i­pants of 0.11 points. Due to tech­ni­cal is­sues, be­hav­ioral data for one of the 24 par­tic­i­pants was not avail­able, there­fore be­hav­ioral per­for­mance was com­puted based on the data from the re­main­ing 23 par­tic­i­pants.

Decoding of se­lec­tive au­di­tory at­ten­tion in a dy­namic switch­ing sce­nario

Participants’ at­ten­tion was de­coded with a back­ward TRF analy­sis, de­scrib­ing the re­la­tion­ship be­tween the EEG sig­nals and the en­ve­lope of the tar­get speech. For each left-out trial, the speech en­ve­lope re­con­structed from the tar­get de­cod­ing model was cor­re­lated with the en­velopes of both the left and right speech streams. Attention was clas­si­fied by de­ter­min­ing which speech stream’s en­ve­lope showed a higher cor­re­la­tion with the re­con­structed en­ve­lope. Since this was a dy­namic at­ten­tion-switch­ing sce­nario, the at­tended speech could al­ter­na­tively cor­re­spond to the left or the right stream. Classification was con­sid­ered cor­rect when the re­con­structed en­ve­lope cor­re­lated more strongly with the tar­get speech en­ve­lope than with the masker en­ve­lope. Classification ac­cu­racy was then com­puted as the pro­por­tion of in­stances where this cri­te­rion was met. To es­tab­lish chance per­for­mance, left and right la­bels were ran­domly shuf­fled 100 times for each de­cod­ing win­dow. As shown in Fig 1C, the longer de­cod­ing win­dows led to higher clas­si­fi­ca­tion per­for­mances. However, even with a 1-second win­dow, clas­si­fi­ca­tion ac­cu­racy was sig­nif­i­cantly above chance level, and all de­cod­ing win­dows yielded clas­si­fi­ca­tion rates sig­nif­i­cantly above the 95th per­centile of its chance dis­tri­b­u­tion (paired two-tailed t test, FDR-corrected for mul­ti­ple com­par­isons for win­dows of 1 s, 2 s, 4 s, 8 s, 16 s, 32 s, re­spec­tively: p = 0.47e−9; 0.53e−9; 0.27e−9; 0.24e−9; 0.24e−9; 0.24e−9). These find­ings align with pre­vi­ous work de­cod­ing sus­tained at­ten­tion or em­ploy­ing match-vs-mis­match clas­si­fi­ca­tion met­rics [9,31,36], and con­firm that a clas­si­fi­ca­tion based on the en­ve­lope re­con­struc­tion can re­li­ably track se­lec­tive at­ten­tion even dur­ing at­ten­tion switches.

Neural track­ing of com­pet­ing speech streams in a dy­namic switch­ing sce­nario re­flects the lis­ten­er’s fo­cus of at­ten­tion and is re­lated to changes in al­pha ERSP

A mul­ti­vari­ate TRF analy­sis was car­ried out to char­ac­ter­ize the neural track­ing of two com­pet­ing speech streams in a set­ting where par­tic­i­pants were in­structed to dy­nam­i­cally switch their at­ten­tion be­tween the two streams. Single-subject TRFs were trained on the tar­get stream and tested on both speak­ers (i.e., Spk1 and Spk2) us­ing a mul­ti­vari­ate speech rep­re­sen­ta­tion that in­cluded Envelope, Word Onset and Word Surprisal fea­tures (for more de­tails see Methods). EEG pre­dic­tion cor­re­la­tions were com­puted us­ing a slid­ing win­dow to cor­re­late true and pre­dicted EEG sig­nals over time, with a leave-one-out cross-val­i­da­tion pro­ce­dure and were av­er­aged across all EEG chan­nels. Importantly, be­cause these cor­re­la­tions are com­puted us­ing slid­ing win­dows, the re­sult­ing switch tim­ing de­pends on the slid­ing win­dow length. As such, the tem­po­ral dy­nam­ics de­riv­ing from our analy­ses do not re­flect the ex­act tim­ing of the un­der­ly­ing neural processes, and they should al­ways be in­ter­preted with the caveat of the slid­ing win­dow length.

In or­der to an­a­lyze ro­bust switch­ing dy­nam­ics, we se­lected 21 par­tic­i­pants dis­play­ing a re­li­able at­ten­tional bias over the course of the switch, based on an above-chance clas­si­fi­ca­tion ac­cu­racy cri­te­rion (>50%) over the course of the switch. In do­ing so, we re­moved par­tic­i­pants for whom the start and end points of the (dis)engagement could not be es­ti­mated (note that this ex­clu­sion is de­ter­mined be­fore iden­ti­fy­ing the start/​end es­ti­mates; in that sense, this is dif­fer­ent from an out­lier re­moval, which would ex­clude ex­treme start/​end val­ues in­stead).

Aligning with our ex­pec­ta­tion (Fig 2A), EEG pre­dic­tion cor­re­la­tions around the switch­ing cue re­flected track­ing of Spk1 and Spk2 streams con­sis­tent with the at­ten­tion in­struc­tions, such that Spk1 was sig­nif­i­cantly more tracked than Spk2 be­fore the switch, while the re­verse pat­tern was ob­served af­ter the switch (paired two-tailed t test of Spk1-Spk2 dif­fer­ence against zero, FDR-corrected for mul­ti­ple com­par­isons, p < 0.005).

As the at­ten­tion switch un­folds, also the grand-mean ERSP in the al­pha fre­quency band dis­played a sta­tis­ti­cally sig­nif­i­cant change com­pared to base­line (one-sample t test against zero, FDR-corrected for mul­ti­ple com­par­isons), re­veal­ing a pat­tern of oc­cip­ito-pari­etal neg­a­tiv­ity in the scalp topogra­phies (Fig 2B). This is con­sis­tent with our ex­pec­ta­tion of an im­pact of at­ten­tional re­ori­en­ta­tion on the EEG al­pha band, which has al­ready been shown to re­flect at­ten­tion switch­ing be­hav­ior in com­pet­ing speech lis­ten­ing sce­nar­ios [29].

EEG pre­dic­tion cor­re­la­tions for Spk1 and Spk2 con­verged, be­fore sig­nif­i­cantly sep­a­rat­ing again once the switch­ing process was con­cluded and, pre­sum­ably, the at­ten­tion was fully re­al­lo­cated. Here, we re­fer to the time point when EEG pre­dic­tion over­laps be­tween Spk1 and Spk2 as the en­cod­ing switch point. Given the ob­served sta­tis­ti­cally sig­nif­i­cant drop in al­pha ERSP, we asked how the tem­po­ral dy­nam­ics of this drop com­pared to those of the EEG pre­dic­tion cor­re­la­tions. To ad­dress this, for each par­tic­i­pant, we iden­ti­fied the time of the al­pha ERSP min­i­mum, and the en­cod­ing switch point, based on an en­cod­ing win­dow of 4s (Fig 2C). The choice of this par­tic­u­lar en­cod­ing win­dow for our main analy­sis is jus­ti­fied based on the clas­si­fi­ca­tion ac­cu­racy re­sults (Fig 1C), since it is a good com­pro­mise be­tween tem­po­ral res­o­lu­tion and clas­si­fi­ca­tion per­for­mance. However, the same pat­tern of re­sults holds when con­sid­er­ing mul­ti­ple en­cod­ing win­dows si­mul­ta­ne­ously (S1 Fig). A paired t test com­par­ing the tem­po­ral dy­nam­ics of the al­pha ERSP and the EEG pre­dic­tion cor­re­la­tions showed that the min­i­mum of the al­pha ERSP drop sig­nif­i­cantly fol­lows the en­cod­ing switch point (t(20) = 4.29, p = 3.59e-4, Cohen’s d = 0.94).

We then eval­u­ated mul­ti­ple en­cod­ing win­dow lengths, as­sess­ing the ef­fect of Metric (encoding switch ver­sus ERSP min­i­mum) and Window (1, 2, 4, 8 s) on the tim­ing of the en­cod­ing switch and the min­i­mum of the al­pha ERSP with a 2-way re­peated mea­sures ANOVA. The analy­sis re­vealed that the tem­po­ral dy­nam­ics of both en­cod­ing switch and al­pha ERSP min­i­mum be­came longer as the en­cod­ing win­dow length in­creased (F(1.68, 33.55) = 52.77, p = 2.3e−10, ηp2 = 0.72; a Greenhouse-Geisser’s cor­rec­tion ap­plied due to spheric­ity vi­o­la­tion), which is un­sur­pris­ing given the method­olog­i­cal con­straints we dis­cussed (see Methods). More in­ter­est­ingly for our ques­tion, a sta­tis­ti­cally sig­nif­i­cant ef­fect of Metric emerged (F(1,20) = 20.26, p = 2.18e−4, ηp2 = 0.5), with the al­pha ERSP min­i­mum oc­cur­ring sig­nif­i­cantly later than the en­cod­ing switch across a range of en­cod­ing win­dows (Holm-corrected post-hoc t test: t(20) = 4.5, p = 2.18e−4, Cohen’s d = 0.97).

Dissecting the tem­po­ral dy­nam­ics of at­ten­tional dis­en­gage­ment and en­gage­ment dur­ing at­ten­tion switch­ing

The at­ten­tion switch­ing cue prompts the lis­tener to re­al­lo­cate their at­ten­tion from the pre­vi­ously at­tended speaker, Spk1, to the newly at­tended speaker, Spk2. While this re-rout­ing of at­ten­tion ap­pears to be a sin­gle, uni­fied process, it is pos­si­ble to dis­tin­guish two sep­a­rate op­er­a­tions that are nec­es­sary for it to hap­pen: dis­en­gage­ment, which we de­fine as the de­crease in neural track­ing for the pre­vi­ously at­tended speech stream, and en­gage­ment, which we de­fine as the in­crease in neural track­ing for the pre­vi­ously un­at­tended speech stream. Our goal was to clar­ify the tem­po­ral dy­nam­ics of these two op­er­a­tions to un­der­stand whether they oc­cur fully in par­al­lel, se­ri­ally, or with a cer­tain de­gree of over­lap. It is worth not­ing that, due to the use of slid­ing en­cod­ing win­dows, the es­ti­mated tem­po­ral dy­nam­ics of en­gage­ment and dis­en­gage­ment do not re­flect the ex­act time course of the un­der­ly­ing neural processes and should be in­ter­preted as rel­a­tive, rather than ab­solute, tem­po­ral met­rics. As in the pre­vi­ous analy­sis, we first se­lected par­tic­i­pants dis­play­ing a re­li­able at­ten­tional bias over the course of the switch (see Methods). For this se­lec­tion of 21 par­tic­i­pants, we fit­ted a piece­wise lin­ear re­gres­sion on sin­gle-sub­ject EEG pre­dic­tion cor­re­la­tions, and found the op­ti­mal break­points, cor­re­spond­ing to the start and end time points of dis­en­gage­ment and en­gage­ment (Fig 2C). As above, we chose to fo­cus on an ex­am­ple win­dow of 4 s and later repli­cated our re­sults on a range of en­cod­ing win­dow lengths. To fur­ther char­ac­ter­ize the spa­tial pat­terns of en­gage­ment and dis­en­gage­ment processes, scalp topogra­phies of the EEG pre­dic­tion cor­re­la­tions at se­lected time points are shown in S2 Fig, in­di­cat­ing that the most pre­dic­tive chan­nels were pre­dom­i­nantly lo­cated over cen­tral-pari­etal re­gions. Disengagement and en­gage­ment processes were com­pared sep­a­rately based on their start times and end times, re­veal­ing con­sis­tently ear­lier tem­po­ral dy­nam­ics for the en­gage­ment com­pared to the dis­en­gage­ment. Engagement to the newly at­tended speaker started sig­nif­i­cantly ear­lier than the dis­en­gage­ment from the pre­vi­ously at­tended speaker (paired-sample t test: t(20) = 2.37, p = 0.03, Cohen’s d = 0.52), and fin­ished sig­nif­i­cantly ear­lier (paired-sample t test: t(20) = 2.35, p = 0.03, Cohen’s d = 0.39).

We then ex­tended our analy­sis to a range of slid­ing win­dow lengths and com­pared start times and end times for dis­en­gage­ment and en­gage­ment processes, in­clud­ing Window (1, 2, 4, 8, 16 s) and Process (disengagement ver­sus en­gage­ment) as main fac­tors in a re­peated mea­sures ANOVA. Regarding the start points (Fig 3A), our analy­ses re­vealed an ex­pected sta­tis­ti­cally sig­nif­i­cant ef­fect of Window (F(1.51,30.14) = 9.7, p = 0.001, ηp2 = 0.33; the as­sump­tion of spheric­ity was not met; hence, a Greenhouse–Geisser’s cor­rec­tion was ap­plied), with longer tem­po­ral dy­nam­ics cor­re­spond­ing to longer en­cod­ing win­dow lengths. More im­por­tantly, we also ob­served a sig­nif­i­cant main ef­fect of Process (F(1,20) = 5.48, p = 0.03, ηp2 = 0.21), with en­gage­ment to the newly at­tended stream start­ing sig­nif­i­cantly ear­lier than the dis­en­gage­ment to the pre­vi­ously at­tended stream (Holm-corrected post-hoc t test: t(20) = 2.34, p = 0.03, Cohen’s d = 0.54). The same sta­tis­ti­cal analy­sis was re­peated sep­a­rately on the end time points of dis­en­gage­ment and en­gage­ment processes (Fig 3B), re­veal­ing once again a main ef­fect of Window, whereby longer en­cod­ing win­dows yield longer tem­po­ral tran­si­tions (F(1.97,39.32) = 31.76, p = 7.2e−9, ηp2 = 0.61; the as­sump­tion of spheric­ity was not met; hence, a Greenhouse–Geisser’s cor­rec­tion was ap­plied). A sig­nif­i­cant main ef­fect of Process also emerged (F(1,20) = 4.46, p = 0.047, ηp2 = 0.18), re­veal­ing that the process of en­gage­ment to the newly at­tended speaker, not only starts, but also ends sig­nif­i­cantly ear­lier than the dis­en­gage­ment (Holm-corrected post-hoc t test: t(20) = 2.11, p = 0.047, Cohen’s d = 0.58).

A fol­low-up analy­sis in­clud­ing the three par­tic­i­pants with lower-than-chance clas­si­fi­ca­tion ac­cu­racy around the switch­ing cue con­firmed that these data points in­tro­duced noise to the es­ti­ma­tion of en­gage­ment and dis­en­gage­ment la­ten­cies. This was ex­pected, as the start and end tran­si­tion points can­not be de­ter­mined in those par­tic­i­pants. The pat­terns ob­served were qual­i­ta­tively sim­i­lar to the main re­sult re­ported above, with ear­lier tem­po­ral dy­nam­ics for the en­gage­ment com­pared to the dis­en­gage­ment, al­beit with weaker ef­fects be­low the sta­tis­ti­cal sig­nif­i­cance thresh­old (repeated-measures ANOVA; start point: F(1,23) = 2.69, p = 0.11, ηp2 = 0.1; end point: F(1,23) = 2.96, p = 0.1, ηp2 = 0.11).

Determining how lex­i­cal pre­dic­tions are built dur­ing at­ten­tion switch­ing

Reorienting at­ten­tion to a dif­fer­ent speech stream im­plies a change of con­text and, con­se­quently, dif­fer­ent se­man­tic pri­ors for lex­i­cal pre­dic­tions. We thus hy­poth­e­sized that in­cor­po­rat­ing this change of con­text into the struc­ture of our se­man­tic re­gres­sor in a mul­ti­vari­ate en­cod­ing TRF model would in­crease EEG pre­dic­tion cor­re­la­tions, as it would bet­ter re­flect the dy­nam­i­cally up­dat­ing neural track­ing of the com­pet­ing speech streams. We com­pared four al­ter­na­tive mod­els rep­re­sent­ing how con­text could be in­cre­men­tally ac­cu­mu­lated for per­form­ing lex­i­cal pre­dic­tions at one par­tic­u­lar at­ten­tion block (e.g., B7, in Fig 4A). A naïve Oracle model, which uses all avail­able con­text of pre­vi­ous blocks from the cur­rent stream, whether at­tended or un­at­tended, to pre­dict words from the cur­rent block, served as our base­line, since it was es­sen­tially a switch-un­aware con­tex­tual rep­re­sen­ta­tion. Speaker-Specific and Attention mod­els were in­stead switch-aware mod­els, as they only con­sid­ered pre­vi­ously at­tended blocks as part of the con­text for lex­i­cal pre­dic­tions. Speaker-Specific as­sumed a higher de­gree of stream seg­re­ga­tion, since its con­text only con­sisted of pre­vi­ously at­tended blocks from the same speech stream, while Attention in­cluded any pre­vi­ously at­tended block from both streams. The Reset model in­stead ig­nored all pre­vi­ously at­tended blocks from any of the streams and com­puted con­text only over the course of the cur­rent block of at­ten­tion, as if the pri­ors for lex­i­cal pre­dic­tions were re­set at each at­ten­tion switch (Fig 4A).

As lex­i­cal en­tropy is a proxy of un­cer­tainty for next-word pre­dic­tion, its val­ues should be im­pacted by a switch­ing cue, which de­ter­mines an abrupt change of con­text. Fig 4B shows the change of av­er­age lex­i­cal en­tropy val­ues in words pre­ced­ing and fol­low­ing the switch cue, which vary de­pend­ing on the con­text mod­els. It can be ob­served that the Reset model peaks with the high­est un­cer­tainty and slowly de­cays over the course of the next words, while the Attention and Speaker Specific mod­els have over­all sim­i­lar lex­i­cal en­tropy dy­nam­ics and more sta­ble val­ues. Consistently with its switch-un­aware na­ture, the Oracle model in­stead dis­plays en­tropy val­ues that are largely un­changed de­spite the switch. An ex­plicit com­par­i­son of the av­er­age en­tropy val­ues of the four con­text-ac­cu­mu­la­tion strate­gies re­vealed sta­tis­ti­cally sig­nif­i­cant dif­fer­ences (repeated-measures ANOVA, Greenhouse-Geisser cor­rec­tion due to spheric­ity vi­o­la­tion; F(1,19) = 39.57, p = 9.59e−10, ηp2 = 0.68). Post-hoc pair­wise tests (Holm-adjusted) in­di­cated that the Reset model showed an in­ter­me­di­ate av­er­age en­tropy, sig­nif­i­cantly higher than the Oracle model (t(19) = 5.75, p = 1.44e−6, Cohen’s d = 0.45), and sig­nif­i­cantly lower than the Attention (t(19) = 4.64, p = 6.28e − 5, Cohen’s d = 0.36) and Speaker-Specific (t(19) = 2.24, p = 0.04, Cohen’s d = 0.17) mod­els. As such, de­spite show­ing the high­est peak fol­low­ing the at­ten­tion switch­ing cue, the Reset model had over­all in­ter­me­di­ate en­tropy val­ues across the four lex­i­cal ex­pec­ta­tion mod­els con­sid­ered here.

Lexical sur­prisal and lex­i­cal en­tropy were used as se­man­tic in­for­ma­tion re­gres­sors for each con­text model and sep­a­rately in­cluded in a mul­ti­vari­ate stim­u­lus rep­re­sen­ta­tion to fit sin­gle-sub­ject en­cod­ing TRFs (Envelope-Word Onset-Word Surprisal and Envelope-Word Onset-Word Entropy). Resulting TRF weights and EEG pre­dic­tion cor­re­la­tions were then com­pared across con­text mod­els, with the hy­poth­e­sis that switch-aware and con­text-rich rep­re­sen­ta­tions (e.g., Speaker-Specific or Attention) would best de­scribe neural ac­tiv­ity in at­ten­tion-switch­ing sce­nar­ios.

Before com­par­ing the con­text mod­els, we first tested whether each of them yielded a sig­nif­i­cant en­cod­ing ac­cu­racy gain com­pared to the base­line model only con­sist­ing of acoustic fea­tures (Envelope and Word Onset). When us­ing en­tropy as a re­gres­sor for se­man­tics, all mod­els, with the ex­cep­tion of Oracle, showed a sta­tis­ti­cally sig­nif­i­cant gain, sug­gest­ing a ro­bust track­ing of se­man­tic in­for­ma­tion in ad­di­tion to the stim­u­lus acoustics (paired t-tests: Oracle ver­sus Acoustics: p = 0.2; Spk.Spec. ver­sus Acoustics: p = 0.04; Attention ver­sus Acoustics: p = 0.04; Reset ver­sus Acoustics: p = 0.002). Employing word sur­prisal as se­man­tic re­gres­sor yielded sim­i­lar re­sults, with all the mod­els show­ing a ro­bust en­cod­ing of se­man­tic in­for­ma­tion, apart from Oracle (paired t-tests: Oracle ver­sus Acoustics: p = 0.15; Spk.Spec. ver­sus Acoustics: p = 0.02; Attention ver­sus Acoustics: p = 0.02; Reset ver­sus Acoustics: p = 0.01). The non-sig­nif­i­cant gain of the Oracle model com­pared to the acoustic model was ex­pected, since Oracle was de­signed as a con­trol switch-un­aware model.

In con­trast to our ex­pec­ta­tion, the Reset con­text model was shown to yield higher EEG pre­dic­tion cor­re­la­tion val­ues when en­tropy was used as a re­gres­sor for se­man­tics (Fig 4C). A re­peated mea­sures ANOVA re­vealed a sta­tis­ti­cally sig­nif­i­cant ef­fect of the main fac­tor, Context Model (F(2.1,47.75) = 9, p = 4e−4, ηp2 = 0.28; with Greenhouse-Geisser’s cor­rec­tion). In the Holm-corrected post-hoc tests, the Reset model was shown to yield sig­nif­i­cantly higher en­cod­ing ac­cu­ra­cies than Oracle (t(23) = 4.99, p = 2.63e−5, Cohen’s d = 0.14), Speaker Specific (t(23) = 3.73, p = 0.002, Cohen’s d = 0.1), and Attention (t(23) = 3.28, p = 0.006, Cohen’s d = 0.09). We then as­sessed the dif­fer­ence of TRF weights for the en­tropy fea­ture across the four con­text mod­els (Fig 4D), av­er­ag­ing the weights’ am­pli­tude within a win­dow broadly cen­tered around the TRF-N400 la­tency (350 – 550 ms). A re­peated mea­sure ANOVA was run on the weights’ am­pli­tude val­ues, re­veal­ing a main ef­fect of Context Model (F(3,69) = 15.51, p = 8.2e−8, ηp2 = 0.4). Post-hoc tests (Holm-corrected) showed that weights for the Reset model had lower TRF-N400 am­pli­tude com­pared to Oracle (t(23) = −5.56, p = 2.4e−6, Cohen’s d = 0.45), Attention (t(23) = −5.84, p = 9.2e−7, Cohen’s d = 0.47), and Speaker Specific (t(23) = −5.24, p = 6.5e−6, Cohen’s d = 0.43).

When fit­ting a mul­ti­vari­ate TRF in­clud­ing lex­i­cal sur­prisal as a se­man­tic re­gres­sor, we ob­served a sta­tis­ti­cally sig­nif­i­cant dif­fer­ence in EEG pre­dic­tion cor­re­la­tions be­tween the four con­text mod­els (F(1.59,36.6) = 3.96, p = 0.04, ηp2 = 0.15, with Greenhouse–Geisser cor­rec­tion). Post-hoc analy­ses in­di­cated a sta­tis­ti­cally sig­nif­i­cant dif­fer­ence be­tween the Reset and Oracle mod­els (t(23) = 3.18, p = 0.013, Cohen’s d = 0.1), while all other post-hoc pair­wise com­par­isons did not reach the sig­nif­i­cance thresh­old (p < 0.05). Similarly, no sta­tis­ti­cally sig­nif­i­cant dif­fer­ence emerged when com­par­ing the TRF-N400 am­pli­tude of the mod­els’ TRF weights.

Discussion

Speech com­mu­ni­ca­tion in multi-talker en­vi­ron­ments re­quires a skill­ful com­bi­na­tion of sus­tained at­ten­tion and rapid at­ten­tion switch­ing abil­i­ties [5,30]. While the neu­ro­phys­i­ol­ogy of sus­tained speech at­ten­tion has been widely stud­ied [1,9,37,38], less is known about the neural mech­a­nisms of at­ten­tion switch­ing. Here, we fill this gap with a tai­lored EEG ex­per­i­ment ex­am­in­ing the neu­ro­phys­i­ol­ogy of at­ten­tion switch­ing across dif­fer­ent lev­els of speech ab­strac­tion. In do­ing so, we (1) demon­strated an ex­per­i­men­tal par­a­digm that can suc­cess­fully probe both sus­tained at­ten­tion and at­ten­tion switch­ing mech­a­nisms; (2) suc­cess­fully dis­sected dis­en­gage­ment and en­gage­ment processes with a high tem­po­ral res­o­lu­tion, iden­ti­fy­ing sub­stan­tial asym­me­tries in their tem­po­ral un­fold­ing and a tran­sient si­mul­ta­ne­ous en­cod­ing of two speech streams; and (3) pro­posed a neu­ro­phys­i­o­log­i­cally plau­si­ble ex­pla­na­tion of how our brains up­date and use lex­i­cal con­text when switch­ing at­ten­tion.

The find­ings in this study have sev­eral im­pli­ca­tions for our un­der­stand­ing of speech at­ten­tion switch­ing mech­a­nisms. The asym­me­try mea­sured be­tween dis­en­gage­ment and en­gage­ment processes high­lights the im­por­tance of study­ing the two processes sep­a­rately. That dis­tinc­tion was of­ten not con­sid­ered in pre­vi­ous stud­ies on sus­tained at­ten­tion, which of­ten fo­cused on mea­sures of at­ten­tion bias or clas­si­fi­ca­tion [10,31,34,39]. The ef­fec­tive­ness of such de­cod­ing met­rics has been a dri­ving force for re­search on brain-com­puter in­ter­faces such as cog­ni­tively-con­trolled hear­ing de­vices [40 – 43]. Our find­ing high­lights that en­cod­ing met­rics en­able a suf­fi­cient level of de­tail for dis­en­tan­gling how the en­cod­ing of dif­fer­ent streams evolves over time. Here, we mea­sured an asym­me­try be­tween dis­en­gage­ment and en­gage­ment processes dur­ing at­ten­tion switch­ing in a very spe­cific sce­nario. Indeed, it will be im­por­tant to de­ter­mine how that re­la­tion­ship is mod­u­lated by fac­tors such as cog­ni­tive load, ag­ing, cog­ni­tive abil­i­ties, hear­ing dif­fi­cul­ties, in­ter­est in the speech con­tent, fre­quency of at­ten­tion switches in a trial, among many oth­ers. Of course, fu­ture work should also scru­ti­nize how the spe­cific na­ture of the task might im­pact that phe­nom­e­non.

Sustained at­ten­tion tasks, where par­tic­i­pants fo­cus on a tar­get speech while ig­nor­ing the masker [2,44], in­volve a quite par­tic­u­lar sce­nario where lis­ten­ers have no in­cen­tive to mon­i­tor un­at­tended streams. In real-life sit­u­a­tions, how­ever, lis­ten­ers may have rea­sons to ex­plore al­ter­na­tive speech streams, for ex­am­ple, due to a lack of in­ter­est in the cur­rent speaker. Our ex­per­i­men­tal par­a­digm more closely mir­rors this sce­nario. Although the in­structed na­ture of the task makes it less re­al­is­tic, the par­a­digm in­cen­tivises mon­i­tor­ing the masker and be­ing ready to rapidly switch at­ten­tion, which con­trasts with sus­tained at­ten­tion tasks. While the asym­me­try ob­served may be spe­cific to this ex­per­i­men­tal par­a­digm, the re­sult in­di­cates that our brains can en­gage with a new tar­get even be­fore start­ing the dis­en­gage­ment from the pre­vi­ous one, lead­ing to a tran­sient si­mul­ta­ne­ous track­ing of the two streams, com­pat­i­ble with au­di­tory scene mon­i­tor­ing mech­a­nisms [45,46]. In other words, there is a brief pe­riod, fol­low­ing an at­ten­tion switch, where the track­ing of a new stream be­gins to emerge with­out al­ter­ing the track­ing of the pre­vi­ous stream. The en­gage­ment and dis­en­gage­ment la­ten­cies are win­dow-de­pen­dent and, as such, are not in­tended as ab­solute neural tim­ings. Nonetheless, the en­gage­ment-dis­en­gage­ment asym­me­try is ro­bust to the se­lec­tion of the slid­ing-win­dow length (Fig 3) and can­not be ex­plained by the tem­po­ral smooth­ing in­tro­duced by the win­dow or by trial- and par­tic­i­pant-av­er­ag­ing. The tem­po­ral smooth­ing could, in prin­ci­ple, tem­po­rally stretch the en­gage­ment-dis­en­gage­ment dy­nam­ics, but not gen­er­ate an asym­me­try. Future re­search could in­ves­ti­gate the vari­abil­ity across tri­als and par­tic­i­pants to bet­ter char­ac­ter­ize these tem­po­ral dy­nam­ics. Within the dor­sal–ven­tral at­ten­tion frame­work [47,48], at­ten­tion re­ori­ent­ing re­sults from the flex­i­ble in­ter­play of the goal-di­rected dor­sal net­work and the stim­u­lus-dri­ven ven­tral net­work. Our find­ing of an en­gage­ment-dis­en­gage­ment asym­me­try aligns with this view of an in­te­grated process of at­ten­tion re­lease and re­al­lo­ca­tion, with po­ten­tially over­lap­ping neural dy­nam­ics.

Intuitively, main­tain­ing a tran­sient par­al­lel rep­re­sen­ta­tion of mul­ti­ple speech sources dur­ing at­ten­tion switch­ing is an ef­fi­cient neural pro­cess­ing strat­egy. It al­lows the flex­i­bil­ity to switch back to the pre­vi­ous stream, if nec­es­sary, with­out fully com­mit­ting to the newly at­tended speech im­me­di­ately. This phe­nom­e­non sup­ports pre­vi­ous claims that our brains can process speech maskers be­yond the acoustic level, en­cod­ing lin­guis­tic prop­er­ties to some ex­tent [7,8,16]. Unattended speech streams are also rep­re­sented in hu­man cor­ti­cal ac­tiv­ity, with ev­i­dence for lower en­cod­ing strengths or longer time la­ten­cies than the tar­get stream [1,49,50], and with a gra­di­ent of at­ten­tional bias from pri­mary to non­pri­mary au­di­tory cor­tex [13,14], whereby the un­at­tended stream en­cod­ing tends to be sub­stan­tially re­duced or not mea­sur­able in higher-or­der cor­ti­cal ar­eas [51,52]. Prior re­search has shown that not only the speech en­ve­lope, but also other key fea­tures of the un­at­tended speech, such as acoustic on­sets, are neu­rally rep­re­sented [53,54] and, when not read­ily avail­able due to speech mask­ing, they are even re­stored at later tem­po­ral scales [55]. One in­ter­pre­ta­tion is that en­cod­ing a tem­plate struc­ture of the un­at­tended speech might be a use­ful strat­egy to sup­press it [53]. Other re­search found at­ten­tional fluc­tu­a­tions cor­re­spond­ing to the changes in Target-Masker rel­a­tive sound en­ergy, with ev­i­dence that our brains may en­code some pho­netic in­for­ma­tion of un­at­tended streams [56,57]. The en­cod­ing of such un­at­tended stream in­for­ma­tion may be one of the fac­tors fa­cil­i­tat­ing the rapid en­gage­ment dur­ing at­ten­tion switch­ing.

Using al­pha ERSP as in­di­ca­tor of lis­ten­ing ef­fort, this study re­lated per­cep­tual de­mands with at­ten­tion switch­ing dy­nam­ics. A large re­duc­tion in EEG al­pha-band power was mea­sured con­sis­tently about 4.5 s af­ter the at­ten­tion-switch­ing cue. The ERSP tra­jec­tory sug­gests that a strong lis­ten­ing ef­fort per­sists through­out the at­ten­tion switch, with a sub­stan­tial re­duc­tion near the end of the switch (Fig 2A). Interestingly, the trough of the al­pha ERSP ob­served in this study roughly cor­re­sponds to the mo­ment when the new tar­get stream be­comes fully tracked, i.e., when the en­gage­ment process is com­pleted, which is well be­fore com­ple­tion of the dis­en­gage­ment process. Since the at­ten­tion switch­ing process can be deemed com­pleted when cor­ti­cal track­ing mea­sure­ments re­turn to pre-switch lev­els for both streams, this re­sult points to a link be­tween al­pha power and the en­gage­ment process specif­i­cally. Another pos­si­bil­ity is that the al­pha ERSP dy­nam­ics re­flect a com­bi­na­tion of lis­ten­ing ef­fort while re­fo­cus­ing at­ten­tion on the new tar­get stream and ac­tive sup­pres­sion of the new masker stream. When the newly at­tended stream is tracked at pre-switch lev­els, a suf­fi­cient acoustic and lin­guis­tic con­text on that stream may have been ac­cu­mu­lated to fa­cil­i­tate the track­ing, re­leas­ing cog­ni­tive re­sources. Future stud­ies could ex­plore this pos­si­bil­ity by ex­am­in­ing how the switch dif­fi­culty in­flu­ences the cor­re­spon­dence be­tween cor­ti­cal track­ing asym­me­try and al­pha ERSP. This find­ing ex­tends prior re­search on the neural cor­re­lates of se­lec­tive at­ten­tion and lis­ten­ing ef­fort. Variations in al­pha-band ac­tiv­ity have been as­so­ci­ated not only with au­di­tory at­ten­tion ef­fort [58 – 62], but also with vari­a­tions of in­ter­nally- ver­sus ex­ter­nally-fo­cused brain states [63], and with the ac­tive sup­pres­sion of ir­rel­e­vant in­for­ma­tion in ac­cor­dance with be­hav­ioral goals [64 – 66]. Notably, cog­ni­tive load and the in­hi­bi­tion of ir­rel­e­vant stim­uli ap­pear to be even more strongly in­flu­enced by at­ten­tion re­ori­ent­ing than by main­tain­ing at­ten­tion on a sin­gle speaker [29,67], high­light­ing the in­creased cog­ni­tive de­mands of switch­ing at­ten­tion. While al­pha power has com­monly been linked to sub­jec­tive lis­ten­ing fa­tigue [58,60] or the sig­nal-to-noise ra­tio (SNR) be­tween at­tended and un­at­tended streams [61], it is less fre­quently as­so­ci­ated with neural mark­ers of speech track­ing [68,69]. In this study, we iden­ti­fied a link be­tween the tem­po­ral dy­nam­ics of neural speech track­ing and lis­ten­ing ef­fort, sug­gest­ing a po­ten­tially valu­able met­ric for fu­ture re­search into at­ten­tion-switch­ing chal­lenges.

While cor­ti­cal track­ing and al­pha ERSP met­rics can be used to in­ves­ti­gate how at­ten­tion is dy­nam­i­cally re­al­lo­cated be­tween com­pet­ing streams, they do not spec­ify how higher-level lin­guis­tic rep­re­sen­ta­tions are up­dated dur­ing the at­ten­tion switch. Considering the im­por­tance of con­text in speech com­pre­hen­sion, it is par­tic­u­larly rel­e­vant to in­ves­ti­gate how it is ad­justed while shift­ing at­ten­tion from one speech stream to an­other. Addressing this ques­tion re­quires fo­cus­ing on neural rep­re­sen­ta­tions with rel­a­tively ex­tended tem­po­ral dy­nam­ics, which can be mean­ing­fully com­pared across com­pet­ing mod­els, and with suf­fi­cient dis­tinc­tion from the speech en­ve­lope track­ing to en­able the iso­la­tion of re­lated neural sig­na­tures. Lexical en­tropy and sur­prisal, with their long-la­tency and widely stud­ied neural sig­na­tures, are an ideal choice for char­ac­ter­iz­ing the im­pact of con­text on at­ten­tion switch­ing. Other in­for­ma­tive speech prop­er­ties such as phonol­ogy or prosody would also be im­por­tant to ex­am­ine; how­ever, their fast tem­po­ral dy­nam­ics chal­lenge the iso­la­tion of their at­ten­tion-switch­ing dy­nam­ics from the speech en­ve­lope track­ing. We there­fore fo­cused on how se­man­tic pre­dic­tions are up­dated fol­low­ing an at­ten­tion switch, mod­el­ing whether lin­guis­tic con­text is main­tained, re­set, or up­dated dur­ing the process.

This study com­pared four con­text-ac­cu­mu­la­tion mod­els that var­ied in their sen­si­tiv­ity to at­ten­tion switches and ac­cess to prior con­text: an Oracle model (context-rich but switch-un­aware), Speaker Specific and Attention mod­els (both con­text-rich and switch-aware but dif­fer­ing in stream se­lec­tiv­ity), and a Reset model (aware of the switch but lim­ited to the cur­rent block’s con­text). For each model, a mul­ti­vari­ate TRF was fit us­ing a se­man­tic re­gres­sor aligned with the re­spec­tive con­text strat­egy. Our data in­di­cates that the Reset model best pre­dicted EEG data, out­per­form­ing mod­els that re­tained past con­text and chal­leng­ing the as­sump­tion that prior se­man­tic in­for­ma­tion aids com­pre­hen­sion dur­ing at­ten­tion switches. This find­ing was un­ex­pected but one of the pos­si­ble out­comes that we had hy­poth­e­sized, and it may sug­gest that lis­ten­ers re­set con­text and re­cal­i­brate their lex­i­cal pre­dic­tions dy­nam­i­cally when switch­ing at­ten­tion to a new stream, in line with find­ings in the episodic mem­ory and event-seg­men­ta­tion lit­er­a­ture [70 – 72].

Interestingly, the mul­ti­vari­ate TRF analy­sis re­vealed fine-grained dif­fer­ences among the four con­text mod­els when lex­i­cal en­tropy is used as the se­man­tic re­gres­sor, but not for lex­i­cal sur­prisal. This may re­flect the for­ward-look­ing na­ture of en­tropy, as op­posed to the re­ac­tive na­ture of lex­i­cal sur­prisal. Another con­sid­er­a­tion is that par­tic­i­pants ac­tively per­formed the at­ten­tion switch, there­fore ex­pect­ing to en­counter dif­fer­ent speech ma­te­r­ial, po­ten­tially damp­en­ing the sur­prisal. This was not the case for the LLM model, which did not re­ceive a switch cue. This pos­si­ble mis­match be­tween LLM and hu­man brain could im­pact sur­prisal but not en­tropy (as that re­flects the process of con­text-build­ing), pro­vid­ing a po­ten­tial ex­pla­na­tion of why the Reset model per­forms best only for the en­tropy re­gres­sor. For in­ter­pret­ing these re­sults, it is also im­por­tant to con­sider that LLMs like Mistral are op­ti­mized for next-word pre­dic­tion, with­out a re­quire­ment of be­ing neu­ro­phys­i­o­log­i­cally plau­si­ble. While one in­ter­pre­ta­tion is that our brain’s lex­i­cal pre­dic­tions are built af­ter re­set­ting the con­text, it is also pos­si­ble that Mistral LLM and our brains deal with these speech dis­con­ti­nu­ities in a dif­fer­ent way al­to­gether. With these caveats in mind, counter to con­sis­tent re­ports of the sim­i­lar­ity be­tween mod­ern LLMs with neu­ro­phys­i­o­log­i­cal ac­tiv­ity [73 – 75], the higher neu­ro­phys­i­o­log­i­cal plau­si­bil­ity of the Reset model sug­gests that prior con­text is not availed of by our brains in the way im­ple­mented by the com­pet­ing mod­els, Attention and Spk-Specific.

Other strate­gies mak­ing use of the prior con­text might also be in place. One pos­si­bil­ity is that switch­ing at­ten­tion prompts a dif­fer­ent use of con­text, for ex­am­ple sum­ma­riz­ing its ab­stract mean­ing, as the gist of the story [16]. As such, there could be value in ex­plor­ing dif­fer­ent strate­gies for con­text rep­re­sen­ta­tion, for ex­am­ple, by em­ploy­ing Large Concept Models [76], which are trained and op­ti­mized for sen­tence pre­dic­tion. This lat­ter pos­si­bil­ity is also sup­ported by re­cent work on the ac­cu­mu­la­tion of lin­guis­tic con­text in LLMs and the hu­man brain [77] while lis­ten­ing to con­tin­u­ous mono­logues, show­ing that LLMs with a lim­ited con­text win­dow (32 to­kens) and with ac­cess to a coarse sum­mary of the pre­vi­ous con­text pre­dict neural ac­tiv­ity bet­ter than LLMs with a higher to­ken-mem­ory.

In sum­mary, this study showed that the process of at­ten­tion switch­ing in a re­al­is­tic multi-talker sce­nario can be in­ves­ti­gated in terms of its en­gage­ment and dis­en­gage­ment com­po­nents, with a tran­sient par­al­lel rep­re­sen­ta­tion of the two streams. We high­light the im­por­tance of re­lat­ing met­rics of neural track­ing of speech with met­rics of lis­ten­ing ef­fort and demon­strate that the lis­ten­ing ef­fort starts de­creas­ing fol­low­ing suc­cess­ful dis­am­bigua­tion of the two streams dur­ing at­ten­tional re-al­lo­ca­tion. Finally, we in­tro­duce an ap­proach for mod­el­ing lex­i­cal con­text of dy­namic at­ten­tion sce­nar­ios, show­ing the sen­si­tiv­ity of trans­former-based lan­guage mod­els to sub­tle dif­fer­ences in con­text ac­cu­mu­la­tion strate­gies. These find­ings have im­pli­ca­tions for fu­ture in­ves­ti­ga­tions into the cor­ti­cal mech­a­nisms of at­ten­tion re-ori­ent­ing and can be em­ployed to high­light dif­fer­ences across di­verse pop­u­la­tions in terms of age and hear­ing lev­els.

Methods

Ethics state­ment

Written in­formed con­sent was ob­tained from the study par­tic­i­pants. The study was con­ducted in ac­cor­dance with the Declaration of Helsinki, and the pro­to­col was ap­proved by the School of Psychology Research Ethics Committee of Trinity College Dublin (ethics ap­proval num­ber: SPREC012023 – 08).

Participants and ex­per­i­men­tal pro­ce­dure

We re­cruited 24 young na­tive English speak­ers (between 18 and 39 years of age) to take part in the study. Participants had nor­mal hear­ing, as per a screen­ing pure tone au­dio­gram from 0.25 Hz to 8 kHz and re­ported no his­tory of neu­ro­log­i­cal or psy­chi­atric dis­or­ders and had nor­mal or cor­rected-to-nor­mal vi­sion.

The ex­per­i­ment sim­u­lated a multi-talker sce­nario (Fig 1A) with a cir­cu­lar ar­ray (1.50m ra­dius) of six loud­speak­ers sur­round­ing the lis­tener (at hor­i­zon­tal an­gles of ±30°, ±112.5°, and ±157.5° rel­a­tive to the par­tic­i­pant). Participants were in­structed to dy­nam­i­cally switch their at­ten­tion be­tween left and right speech streams in the fore­ground, fol­low­ing a vi­sual cue (left- or right-point­ing ar­row) in­di­cat­ing the to-be-at­tended side, which was dis­played at the cen­ter of a screen placed in front of them. While switch­ing their at­ten­tion be­tween the fore­ground streams, they were also asked to ig­nore a 16-talker noise played from the four loud­speak­ers in the back­ground (B1-B4, each of them de­liv­er­ing a 4-talker bab­ble). Frontal streams were pre­sented at 60 dB sound pres­sure level (SPL) each, while each of the noise bab­bles was de­liv­ered at a level of 54dB SPL, re­sult­ing in a 3dB SNR of the fore­ground rel­a­tive to the back­ground.

Participants were pre­sented with 20 tri­als (lasting 180 s each) and had to per­form 6 at­ten­tion switches per trial, oc­cur­ring at semi-ran­dom in­ter­vals (Fig 1B). For this rea­son, blocks of sus­tained at­ten­tion to one par­tic­u­lar speech stream var­ied con­sid­er­ably in du­ra­tion, span­ning from 10 to 30 s. For each trial, a dif­fer­ent male and fe­male speech stream was played from the left and right loud­speak­ers in the fore­ground, coun­ter­bal­anc­ing through­out the ex­per­i­ment for side of pre­sen­ta­tion and start of the at­ten­tion block (i.e., the ex­per­i­ment con­sisted of five sub-blocks with the fol­low­ing trial se­quence: Male Left — Attention Start: Left; Female Left — Attention Start: Left; Male Left — Attention Start: Right; Female Left — Attention Start: Right).

Each trial started with the vi­sual cue point­ing to­wards the to-be-at­tended side and back­ground noise only, fol­lowed by the two fore­ground speech streams start­ing si­mul­ta­ne­ously af­ter 5 s. At the end of each trial, par­tic­i­pants an­swered three mul­ti­ple-choice ques­tions. First, they were pre­sented with a four-al­ter­na­tive forced-choice ques­tion re­gard­ing the con­tent of the at­tended speech stream. As at­ten­tion al­ter­nated be­tween the left and right speech streams over the course of the trial, the ques­tion could be about ei­ther of the two com­pet­ing streams. The sec­ond ques­tion was a bi­nary choice as­sess­ing par­tic­i­pants’ pref­er­ence be­tween the two streams (left or right) based on per­sonal in­ter­est. Finally, a 5-point Likert-scale ques­tion (1: very easy; 5: very hard) was used to quan­tify the per­ceived dif­fi­culty of the at­ten­tion switch­ing task. The ex­per­i­ment flow was self-paced and, to min­i­mize fa­tigue, it in­cluded three manda­tory breaks, each last­ing no less than five min­utes, every fifth trial.

Speech streams were pre­sented at a sam­pling rate of 44.1 kHz, de­liv­ered through a Roland Octa-Capture 10 × 10 sound card (24-bit/192 kHz), and played through six PreSonus Eris 4.5BT loud­speak­ers. Participants’ EEG ac­tiv­ity was recorded us­ing a BioSemi ActiveTwo sys­tem at a sam­pling rate of 512 Hz, from 64 elec­trodes po­si­tioned on a stan­dard cap fol­low­ing the International 10/20 sys­tem. An ac­tive (CMS) and a pas­sive (DRL) elec­trode were used as ref­er­ence for all elec­trodes, and two ad­di­tional elec­trodes were placed on the mas­toids for of­fline ref­er­enc­ing. For 21 of our 24 par­tic­i­pants, we ad­di­tion­ally recorded elec­tro-ocu­log­ra­phy (EOG) and elec­tro-myo­g­ra­phy (EMG). Two elec­trodes were placed on the left and right tem­ples to cap­ture hor­i­zon­tal eye move­ments, and two elec­trodes were po­si­tioned above and be­low the left eye to record ver­ti­cal eye move­ments and blinks. To cap­ture EMG ac­tiv­ity re­lated to head ro­ta­tion, an elec­trode was placed on the left del­toid mus­cle. Please note that ac­tiv­ity from these ex­ter­nal elec­trodes has not been an­a­lyzed as part of this study.

Stimuli

The fore­ground speech stim­uli in­cluded 40 TED Talks cov­er­ing a range of top­ics, with 20 fe­male and 20 male pre­sen­ters, each speak­ing in a va­ri­ety of English ac­cents. All speech streams were root-mean-squared (RMS) nor­mal­ized to re­duce dif­fer­ences be­tween male and fe­male voices. Each of the 4-talker back­ground bab­ble sig­nals was ob­tained by sum­ming the au­dio sig­nals of four sep­a­rate TED talks. The long-term av­er­age spec­trum of the bab­ble noise was then ad­justed to align with the over­all spec­trum of both male and fe­male fore­ground speak­ers, to pre­vent in­con­sis­ten­cies in mask­ing.

EEG data pre­pro­cess­ing

Neural data were an­a­lyzed with cus­tom scripts in MATLAB soft­ware (MathWorks), based on pub­licly avail­able scripts and re­sources shared as part of the CNSP ini­tia­tive (Cognition and Natural Sensory Processing; https://​cn­sp­work­shop.net). Neural sig­nals were first band-pass fil­tered be­tween 0.5 Hz and 8 Hz, us­ing a zero-phase shift Butterworth fil­ters of or­der 4, and then down­sam­pled from 512 to 64 Hz. Spherical spline in­ter­po­la­tion was ap­plied to re­place chan­nels that were three stan­dard de­vi­a­tions away from the mean. EEG was then re-ref­er­enced to the av­er­age of the two mas­toid chan­nels.

Speech fea­tures

The cur­rent study aimed to char­ac­ter­ize the neural track­ing of a dy­namic multi-talker sce­nario by mea­sur­ing the re­la­tion­ship be­tween EEG data and var­i­ous fea­tures of the fore­ground speech stim­uli, re­lated to their acoustic and lex­i­cal prop­er­ties. To model the speech acoustics, the au­dios’ broad­band am­pli­tude en­velopes were ex­tracted by tak­ing the ab­solute of the Hilbert trans­form. In or­der to model the lex­i­cal prop­er­ties of speech, the tran­scribed stim­uli and their au­dios were first au­to­mat­i­cally aligned us­ing the WebMAUS Basic aligner [78 – 80], which iden­ti­fied time­stamps cor­re­spond­ing to the start and end of each word. The re­sult­ing au­to­matic align­ment was saved in the TextGrid for­mat and ad­justed man­u­ally, when nec­es­sary, us­ing the Praat soft­ware [81]. The time stamps were then used to build bi­nary word on­set vec­tors in MATLAB. While bi­nary word on­set vec­tors rep­re­sent in­for­ma­tion re­lated to word seg­men­ta­tion, they can also be mod­u­lated ac­cord­ing to each word’s sur­prisal or en­tropy value to rep­re­sent higher-or­der se­man­tic in­for­ma­tion. Word sur­prisal is a mea­sure of how un­ex­pected a word is given its pre­ced­ing lin­guis­tic con­text, and it can be com­puted us­ing Large Language Models (LLMs), as the neg­a­tive log­a­rithm of the prob­a­bil­ity of that word given the pre­vi­ous con­text. Word en­tropy, on the other hand, mea­sures how un­cer­tain or un­pre­dictable the next word is. Here, we used a pre­trained open-source LLM, Mistral-7B-v0.1 [82] to ex­tract word prob­a­bil­i­ties, and then com­puted lex­i­cal sur­prisal and lex­i­cal en­tropy val­ues for each word.

When con­sid­er­ing the dy­namic na­ture of the at­ten­tion switch­ing task, the de­f­i­n­i­tion of what con­text to in­clude for the cur­rent word pre­dic­tion be­comes a non-triv­ial prob­lem. From the ma­chine per­spec­tive, given a word w be­long­ing to, e.g., the left stream, the LLM would pre­dict it more eas­ily when pro­vided with all the avail­able lin­guis­tic con­text from the left stream. However, from the neural/​be­hav­ioral per­spec­tive, since par­tic­i­pants were flex­i­bly re-ori­ent­ing their at­ten­tion be­tween left and right streams, the op­ti­mal con­text would be im­pacted by the at­ten­tion switch and, po­ten­tially, store in­for­ma­tion of pre­vi­ously at­tended blocks. To com­pare these con­text-ac­cu­mu­la­tion al­ter­na­tives, we rep­re­sented con­text ac­cord­ing to four al­ter­na­tive rep­re­sen­ta­tions. A ma­chine-ideal model, Oracle, con­sid­ers as con­text all words pre­ced­ing the cur­rent word in one par­tic­u­lar stream, whether they were at­tended or un­at­tended. As such, this model is un­aware of the switch of at­ten­tion. Among more neu­rally-plau­si­ble and switch-aware con­text ac­cu­mu­la­tion mod­els, we con­structed the Attention model, which in­cor­po­rates as con­text any pre­vi­ously at­tended block from both left and right speech streams, and the Speaker Specific model, which dis­plays a speech stream bias, whereby only pre­vi­ously at­tended blocks of the same stream are in­cluded as con­text for lex­i­cal pre­dic­tion of the cur­rent block. Similarly to Attention and Speaker Specific, the fourth model, Reset, is switch-aware, but not con­text-aware. In fact, it does not keep track of any pre­vi­ous speech block, nei­ther at­tended nor un­at­tended, and in­stead re­sets the con­text fol­low­ing each at­ten­tion switch cue.

Temporal Response Function (TRF) and analy­sis pro­ce­dure

When peo­ple learn I’m a con­sul­tant, con­ver­sa­tion of­ten pro­ceeds to prob­lems and prob­lem-solv­ing. It’s true that I only get hired well af­ter there is a prob­lem. Typically a prob­lem that has got­ten so lousy that no­body wants to deal with it and it has there­fore be­come worth the trou­ble—of spend­ing time, money, ef­fort, and rep­u­ta­tion—to bring in some­body to sort it out.

That said, I like com­plete­ness. What other re­sponses do I no­tice to prob­lems? (Other than solv­ing them.) I don’t know that any of these are uni­ver­sally good or bad. But I do see peo­ple hav­ing three ad­di­tional re­sponses, and act­ing based on them. These are:

Solving prob­lems (the first re­sponse we think of)

Pushing prob­lems around

Preserving prob­lems

Promoting new prob­lems

Let’s look at each of these three ’P’s in turn.

No. 0001. Pushing prob­lems around

When I was fa­cil­i­tat­ing staff-led con­tin­u­ous im­prove­ment pro­jects, this was the com­mon out­come. Making things bet­ter here by mak­ing them worse there. This is what most prob­lem-solv­ing in medium and large or­ga­ni­za­tions look like, be­cause this is what lo­cal op­ti­miza­tion looks like. This is fine, in a cer­tain sense, and a huge waste of time, in an­other. A key point is to not blame peo­ple for push­ing prob­lems around. They’re play­ing the game in front of them, and play­ing to win. Instead, when you see this hap­pen­ing, look for their boss’s boss and fix the in­cen­tives and sys­tem view there.

No. 0002. Preserving prob­lems

Clay Shirky wrote, in part of a 2010 blog post that is no longer on­line:

Institutions will try to pre­serve the prob­lem to which they are the so­lu­tion.

Kevin Kelly named it ‘The Shirky Principle’ and wrote, also in 2010:

The Shirky Principle de­clares that com­plex so­lu­tions (like a com­pany, or an in­dus­try) can be­come so ded­i­cated to the prob­lem they are the so­lu­tion to, that of­ten they in­ad­ver­tently per­pet­u­ate the prob­lem. … Because of the Shirky Principle, […] progress some­times de­mands that we let go of prob­lems.

A very easy thing to look for when there’s a prob­lem are the peo­ple who de­pend on it. Who’d lose out if the prob­lem were solved? You don’t have to agree with these peo­ple—the ones who pre­serve the very prob­lems you’re work­ing to elim­i­nate. But you had bet­ter know who they are and in­clude them in your plan.

Always ask this. It’s one of Neil Postman’s six ques­tions about tech­nol­ogy (from a 1998 lec­ture):

What prob­lems do we cre­ate by solv­ing this prob­lem?

Jerry Weinberg wrote in one of his books:

Once you elim­i­nate your num­ber one prob­lem, you pro­mote num­ber two.The abil­ity to find the prob­lem in any sit­u­a­tion is the con­sul­tan­t’s best as­set. It’s also the con­sul­tan­t’s oc­cu­pa­tional dis­ease. To be a con­sul­tant, you must de­test prob­lems, but if you can’t live with prob­lems, con­sult­ing will kill you.Does this mean you must give up try­ing to solve prob­lems? Not at all. It means that you must give up the il­lu­sion that you’ll ever fin­ish solv­ing prob­lems. Once you give up that il­lu­sion, you’ll be able to re­lax now and then and let the prob­lems take care of them­selves.Peo­ple who can solve prob­lems do lead bet­ter lives. But peo­ple who can ig­nore prob­lems, when they choose to, live the best lives. If you can’t do both, stay out of con­sult­ing.

The abil­ity to find the prob­lem in any sit­u­a­tion is the con­sul­tan­t’s best as­set. It’s also the con­sul­tan­t’s oc­cu­pa­tional dis­ease. To be a con­sul­tant, you must de­test prob­lems, but if you can’t live with prob­lems, con­sult­ing will kill you.

Does this mean you must give up try­ing to solve prob­lems? Not at all. It means that you must give up the il­lu­sion that you’ll ever fin­ish solv­ing prob­lems. Once you give up that il­lu­sion, you’ll be able to re­lax now and then and let the prob­lems take care of them­selves.

People who can solve prob­lems do lead bet­ter lives. But peo­ple who can ig­nore prob­lems, when they choose to, live the best lives. If you can’t do both, stay out of con­sult­ing.

In my own prac­tice, the pri­mary way of dis­pelling this il­lu­sion is to get a good di­a­gram go­ing so that every­body can see their prob­lems, agree on what they are, and pick a few that are ac­tu­ally worth fix­ing. More on that soon.

Read Next

Open space tech­nol­ogy, prin­ci­ple 4: “When it is over, it is over”

Newer post

Two fa­cil­i­ta­tion meth­ods start­ing with a sin­gle line: Actions|Results & +|Δ

Older post

Quad-rotor drone shots taken low to the ground are dif­fi­cult: GPS al­ti­tude is fairly rough on ac­cu­racy, and ob­sta­cle avoid­ance can get sig­nif­i­cantly more dif­fi­cult ver­sus just fly­ing over the every­thing. Cinema rover drones are less com­mon but do get around a num­ber of these prob­lems, es­pe­cially if the sub­jects are not high off the ground. As a fan of kart­ing, light­weight con­trap­tions and pho­tog­ra­phy this seems like a pretty good pro­ject: Tackle a me­chan­i­cally sta­bi­lized video plat­form, pi­lot it re­motely and cap­ture some out­door ac­tion.

Taking a look at the Chassis

I picked up this from BMI sur­plus [link], dur­ing an ad­ven­ture with Jake Hecla [link] and the mys­te­ri­ous Arsenio [link]. For some quick back-story, BMI sur­plus is this in­cred­i­bly in­ter­est­ing sur­plus em­po­rium in Massachusetts, while they do not have tours, you can pick up items that you pur­chase ahead of time. We were for­tu­nate to get some time to browse in­side the in­cred­i­bly dense ar­rays of ma­chines, gad­gets and giz­mos. A lot of this stuff ap­pears to come from Lincoln Laboratory, the friendly neigh­bor­hood spooks. I was able to hag­gle a bit and pur­chased the mys­tery RC chas­sis for ~50 USD + tax, along with some other items.

There was no ac­tual in­for­ma­tion about this thing, what it was used for, why it had a bizarre Z-Axis lin­ear ac­tu­a­tor bolted to the top. I did some dig­ging but was un­able to find any write-ups, tech­ni­cal pa­pers or any­thing about this ‘thing’. Maybe they were sur­vey­ing, maybe it was a mo­bile de­vice to take im­ages at the door height of a car? No idea

Fortunately the ac­tu­a­tor was a sim­ple DC brushed mo­tor lin­ear ac­tu­a­tor, which was easy to con­trol. Behold a scaled dolly shot of this con­trap­tion in-ac­tion.

There is an an­i­mated video here tag.

While this thing was in­ter­est­ing, there was no way that I was go­ing to even try putting a full sized cam­era gim­bal on this thing, so af­ter a few screws the Z-Axis was re­moved, un­der­neath we see a wa­ter-jet plate on, the most com­i­cally du­bi­ous 4 – 40 stand­offs I have ever seen. 4 inch long 4 – 40 stand-offs is wild.

Hardware Mock Up

For the pur­poses of de­ter­min­ing what this could look like, and if it was go­ing to be too un­wieldy, I opted to tem­porar­ily use the ex­ist­ing mount­ing plate, make a 3D printed adapter and get a hint as to what I’d be work­ing with.

First up was to re­move the han­dle as­sem­bly that’s na­tive to the Movi M10, which in­her­ently flips this whole gim­bal up­side-down. I was some­what con­cerned with how much load­ing would now be on these four M3 screws, but that’s some­thing that fu­ture-Dane needs to re­solve. Shown be­low is the gim­bal, with the mount point shown. Likely, go­ing for­ward I would prob­a­bly end up pick­ing up the cen­tral 4 screws and try and me­chan­i­cally pro­vide some ad­di­tional sup­port.

I used some quick ther­mal in­serts to pick up on the long flat por­tions of the ex­ist­ing chas­sis. By us­ing flat-head screws, I can take up a bit of tol­er­ance mis­match, and when loose it can slide back and for­ward down the top plat­form. This print was a place­holder, but it did let me get a good vi­sual of how tall this whole stack-up would be.

Finally, the first test fit to see what the gim­bal looked like on the chas­sis. While this is a quick mock up it did pro­vide some in­sight as to how quickly the height of the whole as­sem­bly could eas­ily creep up if it was­n’t ac­tively be­ing con­strained. The lower the cam­era and as­sem­bly height, the lower the ef­fec­tive ve­hi­cle cen­ter of grav­ity re­mains, in­creas­ing its sta­bil­ity while help­ing re­duce rollover forces. Making a mock-up may take time but it re­ally lets you skip an it­er­a­tion step as in­stead of an ill-de­fined cad model you now have some­thing to in­ter­act with on the lab bench.

There is an an­i­mated video here tag.

It is huge. I took the op­por­tu­nity to also do some place­ment tests for where the bat­tery mounts could end up. As it was fairly ap­par­ent, us­ing bat­ter­ies as bumpers is not a great plan. The two packs would likely need to lay as low as pos­si­ble but still re­main ac­ces­si­ble for hot-swap­ping two packs at a time. The only rea­son­able spot would be along the sides, while not in­ter­fer­ing with the gim­bal or the ground clear­ance.

New Mechanicals

A more struc­tural gim­bal mount

The floppy alu­minum mounted on some adorable 4 – 40 stand­offs was not go­ing to do it, I needed some­thing much more struc­tural. While the gim­bal it­self is not me­chan­i­cally heavy, its mount point does need to be as low as pos­si­ble to help mit­i­gate flip­ping. Ideally the Gimbal-camera as­sem­bly is re­move-able from the frame, such that i can test and tweak the mo­tor con­trol tun­ing, fend­ers and the like with­out putting every­thing else through hell. Time to fire up the CNC, turn some proper stand­offs and rigidly con­nect to the frame.

The ba­sic plot is to pro­vide a solid foun­da­tion as close as me­chan­i­cally pos­si­ble to the frame for the gim­bal and shock mount. Fortunately the sub-frame base plate is made of alu­minum. I’m go­ing to again opt for stand-of­f’s to el­e­vate the plat­form above the drive mo­tor, but just use some large round-stock to pro­vide a se­cure mount.

There is an an­i­mated video here tag.

After some quick ma­chin­ing, drilling and tap­ping we have our new el­e­vated gim­bal plat­form, with coun­ter­sunk M8 flat head screws. This in­ten­tion­ally barely clears the drive mo­tor, and leaves the cen­tral round part of the spring damper mount re­cessed to keep the Z-offset height as low as pos­si­ble.

The spac­ing of the stand­offs is nom­i­nally tied to the clos­est po­si­tions that I could pick up that were co-pla­nar with­out bump­ing into the mo­tor mount / servo mounts. While this is slightly aft of cen­ter, it does per­mit space for the some­what heavy bat­ter­ies to live, ide­ally re­sult­ing in the to­tal mass bal­ance to­wards the cen­ter.

After ver­i­fy­ing lo­ca­tion a few times, I punched some holes through the chas­sis and used some M8 screws with flanges to firmly at­tach the new plat­form to the chas­sis. These were ini­tially just tight­ened to a few new­ton me­ters, but on the fi­nal as­sem­bly did re­ceive some mild loc­tite to help with vi­bra­tion in­duced loss of ten­sion.

Fenders for Ice Racing

One of the big is­sues with rac­ing on a slushy sur­face is the slush get­ting every­where . I do not have a tra­di­tional chas­sis for this ve­hi­cle, so it’s up to me to fig­ure out how to con­tain the slush, while not be­ing too in­flex­i­ble. This is a great op­tion for 3D printed parts, as it’s a lot of odd shapes and con­tours, how­ever, this is also a bat­tle bots crossover episodes , and oddly, flex­i­ble things should out-per­form sta­tic things. Initially, for it­er­at­ing and quick pro­to­typ­ing, the fend­ers are nor­mal PLA.

Comically, up to this point, I have never ac­tu­ally printed with com­mer­cial TPU, which is the go-to plas­tic flex­i­ble elas­tomer. I cant think of a bet­ter use case than fend­ers on a RC car frame. Just like a com­muter-bike, the more of the wheel path that is cov­ered, the less can get sprayed onto the cam­era and gim­bal. The plot is to pickup a hard mount, which in this case is some alu­minum an­gle stock, use a stan­dard PLA part to pro­vide a mount­ing point close to the wheel, and then tran­si­tion into a TPU part around the wheel.

I had ini­tially tried 85A TPU and it was just a bit too fid­dly to re­li­ably print, with­out re-do­ing my fil­a­ment spool hold­ers to be lower re­sis­tance. Any re­sis­tance was caus­ing un­der-ex­tru­sion and it was dif­fi­cult to man­age. I opted for switch­ing to 95A fil­a­ment, which is stiffer, and did a sub­se­quent slow week­end print to make flex­i­ble tire fend­ers for the rear. The rear mounts are mir­rored but both mate with three M6 threaded flange screws. After fid­dling with set­tings to get a re­li­able print on the fend­ers, its a good idea to start plan­ning a TPU front bumper to help keep this thing from get­ting smashed too eas­ily.

The print time for a sin­gle wheel fender was ap­proach­ing a day and a half on a Prusa MK4. For this spe­cific print, I opted for high wall count 25% in­fill, this should re­sult in a fairly stiff part that’s able to ab­sorb im­pacts.

While it would have been ideal if all four fend­ers were the same part, the steer­ing up front re­quires a lot more clear­ance, re­sult­ing in a larger ra­dius. The printed part does pick up the same hard mount point

There is an an­i­mated video here tag.

I did­n’t men­tion how awk­ward re­mov­ing the sup­port ma­te­r­ial is for large TPU prints. Its time con­sum­ing just due to how im­pact and force ab­sorbent it is. The layer-layer ad­he­sion is amaz­ing.

There is an an­i­mated video here tag.

Finally a spool up of the chas­sis on stand-off blocks, fu­eled by the two se­ries DeWalt bat­ter­ies. As I learned from a friend, full RPM in this sit­u­a­tion is out­side of the mo­tor and drive-train specs, given that i was now run­ning 10S / ~40v. The short blip of full speed was plenty to see how fright­en­ing this mon­ster would be­come.

There is an an­i­mated video here tag.

To mount the side plates, i used a long tip marker to in­di­cate where the print would align with the chas­sis and used a punch to trans­fer those lo­ca­tions so they could be sub­se­quently tapped.

There is an an­i­mated video here tag.

With the holes in the gim­bal riser tapped and the chas­sis clear­ance holes drilled, it’s time to put every­thing to­gether. Due to the right side of the ve­hi­cle’s sim­plic­ity, I’m opt­ing to in­stall its cover plate first, there’s only two wires to worry about. Time to put the elec­tric screw­driver to work. Six M3 screws grab the alu­minum top-plate and six sub­se­quent M3 screws at­tach the bot­tom of the print to the frame.

This video has au­dio: Click to un-mute

Battery Mounts

Both sides of the gim­bal mount­ing plate fit in sep­a­rate, blue, printed plates that hold the main power switch, pre-charge and bat­tery mount points. The prints pick up tapped M3 holes above and con­tain M3 ther­mal in­serts be­low to pickup screws from the chas­sis.

The printed mounts then mate to an off the shelf, in­jec­tion molded DeWalt Battery ter­mi­nal, con­nect­ing with four M4 screws. With the bat­tery latched in, it is sur­pris­ingly abuse tol­er­ant. Shown be­low is a stan­dard DeWalt 6AH 20V bat­tery mod­ule. The or­ange cover cap­tures any ex­posed wiring pre­sent from the bat­tery adapter. Ideally the gap-space gets cov­ered up to help mit­i­gate ice slush ingress, but that’s a prob­lem for fu­ture Dane.

There is an an­i­mated video here tag.

For the left side of the chas­sis cover we have a lot more go­ing on. The main power switch, pre-charge and pack volt­age in­di­ca­tion is pre­sent, with as­so­ci­ated wiring on the back­side. Behind this are the dc/​dc con­vert­ers that pro­vide steer­ing power and in­di­ca­tion light power, along with our re­mote con­trolled re­lay for head­lights / tail lights.

“Headlights and Tail lights”

Having some vi­sual in­di­ca­tion on this con­trap­tion is help­ful, es­pe­cially with how quickly dusk ap­pears. A small head­light and rear fac­ing red lights should be a quick ad­di­tion:

For a head­light i opted for this wa­ter­proof small mod­ule, in­tended as a 3rd party au­to­mo­tive light. It for­tu­nately takes a wide range of op­er­at­ing volt­ages so i can put to use a 15W dc/​dc mod­ule that’s been col­lect­ing dust. For the tail lights I’m also opt­ing to re-use some 12V in­di­ca­tors pur­chased for a pro­ject from ages ago. While they are not in­cred­i­bly bright they are vis­i­ble from a rea­son­able dis­tance.

We do have a num­ber of chan­nels avail­able on this ra­dio, in­clud­ing switches. Having the abil­ity to dis­able light­ing, if it were in­ter­fer­ing with the cam­era or caus­ing re­flec­tions, would be use­ful, es­pe­cially re­motely. To im­ple­ment this I’m opt­ing to go sim­ple, a very ba­sic RC con­trolled re­lay.

With the sim­ple 3d printed brack­ets in­stalled, I was pretty happy with how things turned out.

There is an an­i­mated video here tag.

Sorting out HD FPV

There’s re­ally four op­tions for long range video links at the mo­ment, low res­o­lu­tion low la­tency ana­log, high de­f­i­n­i­tion high price DJI hard­ware, pre­vi­ous gen­er­a­tion niche cin­ema hard­ware and Open Source Build the whole thing so­lu­tions.

As of writ­ing this, the FCC has banned most of DJI’s hard­ware of­fer­ings [Link]. Given that pre­sent-gen­er­a­tion DJI hard­ware is not banned, but fu­ture gen­er­a­tions are, the prices have be­come quite high. For ref­er­ence, the trans­mit­ter alone is 1100 USD. We want some­thing equiv­a­lent that hits all three ideals: low la­tency, high res­o­lu­tion and

Let’s look at the niche cin­ema hard­ware of yes­ter­year and see if any gad­gets are avail­able for low ru­ble.

Enter the CONNEX by Amimon

This gad­get was re­leased in 2015, roughly a decade ago, but the specs are re­ally re­mark­able, es­pe­cially for the time. 1KM of range? -10C op­er­at­ing rat­ing? Not me­chan­i­cally enor­mous? sounds great!

Here’s a quick overview of the fea­tures di­rect from the man­ual. 1KM / 0.6 mile range is fan­tas­tic for that res­o­lu­tion & la­tency, and 5.8ghz an­tenna hard­ware is now fairly easy to come by. Given that we don’t re­ally know the ori­en­ta­tion of the rover in re­la­tion to the pi­lot, we’re stuck with omni an­ten­nas on the rover. The rover, un­like a quadro­tor drone, is also phys­i­cally on the ground, with the an­ten­nas barely 40 cm from the sur­face. Real world tests will likely net a shorter range but this is al­ready an ex­cel­lent start.

This is pretty ex­cel­lent, and they do ap­pear for ~100 – 200$ used on eBay, but Wait, why have I never heard of these things? Amimon was pur­chased by Teradek, who makes a very sim­i­lar item just for 5X the price. Awesome.

I pur­chased a set of trans­mit­ter and ground sta­tion ra­dios from eBay and got to work sort­ing out how I would in­te­grate them to this ve­hi­cle. One of the dan­gers of work­ing with “older” hard­ware is not the hard­ware it’s the sup­port soft­ware.

Narrators voice “there were soft­ware is­sues”

The specs are quite im­pres­sive on pa­per, specif­i­cally the <1ms la­tency . I was also im­pressed by the -10 Celsius rat­ing. On the rover side of the fence we need to pipe Mini HDMI from the cam­era into the “Air Unit” along with ~14V from the Gimbal bat­tery.

A copy of the man­ual for the Connex Amimon is avail­able here [link], with a lo­cal copy here [link]

Lets build a dis­play and FPV re­ceiver mount

Monitors for out­door use can be a bit tricky, you are in­her­ently try­ing to beat the sun. I have been a fairly big fan of Liliput and opted for their 7″ 1800 nit mon­i­tor, it sup­ports 1080P and is cov­ered in 1/4 – 20 mount points. They just work, have a wide in­put volt­age range, and are so much more rugged than mys­tery 4-character ama­zon brands.

The Connex re­ceiver is some­what large and the an­ten­nas do need to be fac­ing up­wards, so let’s stick the whole thing on the back of the Liliput mon­i­tor. There are four M2 threaded holes on the back­side of the Connex re­ceiver, so our part will pick up the sides of the Liliput and pro­vide a place for four screws to mate to the re­ceiver, hug­ging the back of the mon­i­tor. Fortunately, all the in­put/​out­puts of the re­ceiver are on the sides, so as long as we prop­erly me­chan­i­cally con­strain the ca­bling we should have a pretty ex­cel­lent lit­tle setup.

After some it­er­a­tion and fit-test­ing, I came up with a slightly more me­chan­i­cally ro­bust part, us­ing some epox­ied in M3 screws to act as me­chan­i­cal stiff­en­ers on the parts that con­nect to the side of the mon­i­tor. M3 threaded in­serts pro­vide spots to help con­strain the HDMI and power ca­bles, while keep­ing the path to the switch and link con­nec­tors ac­ces­si­ble.

Now that we have a mon­i­tor and re­ceiver for the hand­held con­trols por­tion of this pro­ject. For the ra­dio, I’m opt­ing to use a Taranis X7, mostly be­cause I picked one up at Guardian from the left­over cruft pile. I’m a big fan of the X7, I used it for SnowBot [link]. My only qualm is that there are no places to mount ex­ter­nal gad­gets or giz­mos, if this had some M6 or 1/4 – 20 threaded mount points it would be ex­cel­lent.

Long ago, Guardian Agriculture was Kiwi Agriculture

I needed a ra­dio for re­mote con­trol and planned to just have a portable dis­play that would fol­low along with that re­mote. Yes, as men­tioned by FRED, I could set up a ground sta­tion and a tri­pod but the prob­a­bil­ity of me knock­ing over a tri­pod out in the cold is very high. So let’s start out with the only ac­tual mount point on the X7, the neck-strap mount.

We also have one more ‘hard point’ that we can pick up, the an­tenna pro­tru­sion that’s in­jec­tion molded into the case. If we can pick up a hard point mount there and the neck-strap, and con­tour closely to the case we should be set for at least a first pass at a mon­i­tor mount.

After a num­ber of it­er­a­tions, I ended up with an M6 long ther­mal in­sert to pickup the neck­lace mount and a heav­ily walled hole to pickup the an­tenna mount. This breaks out into one M4 ther­mal in­sert for the mon­i­tor and two aux­il­iary M3 ther­mal in­serts for “whatever sub­se­quent spac­ing mod­i­fi­ca­tions i need”. Note that I opted for high in­fill and high wall-count for this part as the lever arm from the mon­i­tor is quite high.

It did end up work­ing out fairly well, es­pe­cially for a first pass, the mon­i­tor is quite heavy and the stock bendy mount was quite lim­ited, so some more it­er­at­ing to do.

Configuring the Connex Amimon

The nom­i­nal pair­ing process for these two ra­dios is fairly straight­for­ward, and does not re­quire any com­pan­ion soft­ware. To pair the pro­ce­dure is fairly straight­for­ward:

Apply power to air unit

press and hold link for 5 sec­onds, or un­til it starts fast-blink­ing

Apply power to ground sta­tion unit (preferably with a mon­i­tor con­nected)

press and hold link for 5 sec­onds, or un­til it starts fast-blink­ing

Follow the on screen di­rec­tions on the air unit and it will show a progress bar for pair­ing

I fol­lowed these di­rec­tions and alas, no dice. The ground unit sat in pair­ing mode for 5+ min­utes and then timed out.

I did some dig­ging and found the con­fig­u­ra­tion tool. It did “just work” right out of the box, which is great for ~10 yr old soft­ware, but I came upon my first dilemma. The ground unit and the air unit had wildly dif­fer­ent firmware ver­sions . I did at­tempt dif­fer­ent vari­a­tions of the pair­ing pro­ce­dure, but alas each go they see each other but refuse to pair. The soft­ware tool did have an up­date fea­ture, but it was too smart, it na­tively pings a server to check for new, com­pat­i­ble, firmware ver­sions. Those servers are un­for­tu­nately gone. Shown be­low is the “No Server Connection!” mes­sage, also show­ing the mis­match be­tween the Air Unit and the Ground Unit.

Time to find some help. Between 2016 and now, the ini­tial cre­ator of this hard­ware / soft­ware was ab­sorbed into Teledek, which gen­er­ally re­sults in pre­vi­ous hard­ware get­ting shelved. I was able to get in touch with a sup­port en­gi­neer and was given the best sup­port email ever: there’s an of­fline up­date mode

Enabling Offline Update Mode for the Connex Amimon

To en­able of­fline up­date mode here is the pro­ce­dure:

Download the Connex Management Tool

The tool is avail­able from the ven­dor here [Link] and a lo­cal copy is avail­able here [Link]

Extract the man­age­ment tool and in­stall

For the pur­poses of this write up, lets as­sume you are run­ning win­dows

Place a blank file in the pro­gram di­rec­tory

Create a file “local.txt” with no con­tents in the pro­gram folder, C:\Program Files (x86)\Amimon\Connex

Grab the lat­est firmware files

The lat­est firmware files are avail­able here [link] and a backup copy is avail­able here [Link]. Download and save lo­cally

Unzip the firmware, you should end up with all of the firmware op­tions for US, EU and over­seas.

Launch the Connex Software

Click up­date and browse to the cor­rect firmware

With the se­cret of­fline up­date mode, we now have the abil­ity to push these units to the same firmware ver­sion. I started with the air unit and then com­pleted with the ground unit. Shown be­low is the process, un­for­tu­nately OBS screen grab missed the ‘open a win­dow to browse to the ac­tual file’, but for ref­er­ence I used “PR_ID_UAV10100US000_PR_NAME_ConnexUS_VER_4_5_61.amn” as the fi­nal tar­get firmware for both units.

There is an an­i­mated video here tag.

It works!

Propulsion Electronics

This did come with a cas­tle cre­ations mo­tor con­troller, how­ever, I did want some­thing that I was able to ad­just set points for, and have some flex­i­bil­ity go­ing for­ward re­gard­ing au­ton­omy. I opted for a VESC 75V100, nom­i­nally as I had one avail­able and had used one on a pre­vi­ous pro­ject. The 75V100 is very bud­get friendly, and just re­quires some sil­i­cone glue to make it ro­bust enough to sur­vive shock and vibe. Internally there is a large elec­trolytic ca­pac­i­tor that has no me­chan­i­cal con­straints.

For the ini­tial spool-ups, I used a bench sup­ply at 30V, which hon­estly is in­ad­vis­able. Bench sup­plies are not four quad­rant de­vices and re-gen cur­rents from the mo­tor spool­ing down can over-volt the sup­ply and cause is­sues with the con­troller. Nominally I was mostly in­ter­ested in ver­i­fy­ing that the VESC could run the mo­tor sen­sor-less

Fortunately, even with a ba­sic tune we got what we were look­ing for, mo­tor char­ac­ter­is­tics: 5 mOhm phase re­sis­tance and low phase in­duc­tance. With this ba­sic in­for­ma­tion we can do a quick spool up test, mak­ing sure to not spool down quickly.

Hello! I’ve been work­ing on a Django site re­cently, and I de­cided to use SQLite as the data­base. When I was get­ting started with us­ing SQLite as data­base for a web­site I read a bunch of blog posts about how it is to­tally fine to use SQLite in pro­duc­tion for a small site and I think it is to­tally fine, but what I did not fully ap­pre­ci­ate is that SQLite is still a data­base, data­bases are com­pli­cated, and I do not know a lot about op­er­at­ing data­bases.

So here are a cou­ple of small things I’ve been learn­ing about run­ning SQLite. This is the 4th web­site I’ve used SQLite for, and I think this one is harder be­cause with the power of the Django ORM I’ve been mak­ing the data­base do more work than I was pre­vi­ously with­out Django.

I started by turn­ing on WAL mode like all the blog posts said to do and hop­ing for the best.

ANALYZE is ap­par­ently im­por­tant

Today I was run­ning a query (using SQLite’s FTS5 for full-text search) on a table with 4000 rows and it took 5 sec­onds. That seemed wrong to me: com­put­ers are fast!

It turned out that what I needed to do was to run ANALYZE! Immediately the prob­lem query went from tak­ing 5 sec­onds to like 0.05 sec­onds (or some other num­ber small enough that I did­n’t care to in­ves­ti­gate fur­ther). I still don’t know ex­actly what went wrong in the query plan, but my best guess is that it was some sort of ac­ci­den­tally qua­dratic thing.

ANALYZE gen­er­ates “statistics” (I guess about the num­ber of rows in each table? and pre­sum­ably other things?) so that the query plan­ner can make bet­ter choices.

Maybe one day I’ll learn to read a query plan.

clean­ing up the data­base is tricky

Occasionally I’ve run into sit­u­a­tions where I ac­ci­den­tally put a bunch of rows in my data­base that I don’t want to be there (for ex­am­ple com­pleted tasks from django-tasks-db), and I want to clean them up.

What’s hap­pened to me a few times in this case is:

I run some kind of com­mand to clean up the rows

The com­mand takes more than 5 sec­onds, since there are a lot of rows (though I still have some ques­tions about why these DELETE state­ments are so slow hon­estly, maybe there’s a bunch of Python code run­ning in­side a trans­ac­tion, I’m not sure)

One of the other work­ers tries to write the data­base while this is hap­pen­ing, and times out af­ter 5 sec­onds (I have a time­out of 5 sec­onds set)

The worker crashes be­cause it could­n’t write to the data­base and the VM shuts down

My ap­proach so far has been to just do these cleanup op­er­a­tions in small batches so that I don’t need to do data­base queries that take more than 5 sec­onds to run. This whole ex­pe­ri­ence has given me more of an ap­pre­ci­a­tion for why some­one might want to use a “real” data­base like Postgres which can have more than one writer at the same time though.

Maybe in the fu­ture I’ll just take the site down for sched­uled main­te­nance in­stead when I need to do this kind of thing, but I haven’t fig­ured out a work­flow for that yet.

no notes on per­for­mance of ORM queries yet

So far I’ve been us­ing Django’s ORM to make any query I want with­out pay­ing any at­ten­tion at all to query per­for­mance and it’s mostly been go­ing okay other than the ANALYZE thing. The data­base is pretty small (maybe 10000 rows?) and I ex­pect it to stay pretty small for­ever, so I’m hop­ing that that plan will keep work­ing.

back­ing up sqlite

I’ve done SQLite back­ups a cou­ple of ways. I don’t think I’ve ac­tu­ally tested restor­ing from my back­ups but I do usu­ally try to mon­i­tor them with a dead man’s switch.

way 1: restic

sqlite3 /data/calendar.db “VACUUM INTO ‘/tmp/calendar.sqlite’” gzip /tmp/calendar.sqlite

# Upload backup to S3 # Sometimes the backup gets OOM killed and so it stays locked, do an un­lock restic -r s3://s3.ama­zon­aws.com/​some_bucket/ un­lock # Do the backup & prune old back­ups restic -r s3://s3.ama­zon­aws.com/​some_bucket/ backup /tmp/calendar.sqlite.gz restic -r s3://s3.ama­zon­aws.com/​some_bucket/ snap­shots restic -r s3://s3.ama­zon­aws.com/​some_bucket/ for­get -l 1 -H 6 -d 2 -w 2 -m 2 -y 2 restic -r s3://s3.ama­zon­aws.com/​some_bucket/ prune

way 2: litestream

I started try­ing out Litestream re­cently be­cause I felt like do­ing in­cre­men­tal back­ups might be more ef­fi­cient: my restic back­ups were some­times get­ting OOM killed, and I was a bit tired of it. Basically I just write a con­fig file and run:

litestream repli­cate -config litestream.yml

I set re­ten­tion: 400h in my con­fig file in an at­tempt to re­tain some amount of his­tory of the data­base but I have no idea if it works.

I’ve been back­ing up to AWS, which is al­ways a pain be­cause it’s an­noy­ing to nav­i­gate the AWS con­sole to gen­er­ate cre­den­tials. Maybe one day I’ll move away to some other S3-compatible al­ter­na­tive.

you can use mul­ti­ple data­bases

My cur­rent pro­ject only has one data­base, but one trick I used with Mess with DNS was to split the ta­bles into three sep­a­rate data­base files be­cause I did­n’t ac­tu­ally need my ta­bles to be in the same db. I think it was help­ful.

Mess with DNS has been run­ning on SQLite for 4 years now (since 2022) and it’s been great, I think the move from Postgres was a great choice for that pro­ject.

that’s all!

It’s al­ways kind of fun to see how long it takes me to learn sort of ba­sic things about the tech­nolo­gies I’m us­ing. I think I used SQLite for a web pro­ject for the first time in 2022 and I only learned that ANALYZE ex­isted to­day! I imag­ine in a year or two I’ll be learn­ing about some other very ba­sic fea­ture.

some ref­er­ences

Some blog posts I’ve looked at, other than the of­fi­cial docs:

The de­fin­i­tive guide to us­ing Django with SQLite in pro­duc­tion

a gist on sqlite per­for­mance tun­ing