Loading metrics
Loading metrics
Open Access
Peer-reviewed
Research Article
Emina Aličković,
Johannes Zaar,
Alejandro López Valdés ,
Giovanni M. Di Liberto
Competing speech streams are simultaneously represented in the human cortex during attention switching
Sara Carta,
Emina Aličković,
Johannes Zaar,
Alejandro López Valdés,
Giovanni M. Di Liberto
x
Published: July 16, 2026
https://doi.org/10.1371/journal.pbio.3003876
Figures
Abstract
Successful speech communication in multi-talker scenarios requires a skillful combination of sustained attention and rapid attention switching. While the neurophysiology literature offers detailed insights into the neural underpinnings of sustained attention, there remains considerable uncertainty on how attention switching takes place. In this study, using EEG recordings from normal-hearing adults in an immersive multi-talker environment, we measured the neural encoding of two competing speech streams amid background babble. Participants were cued to switch attention between streams every 15 – 30 s. Neural tracking was assessed via Temporal Response Functions (TRF), confirming reliable decoding of attentional focus. Our results indicate asymmetric disengagement and engagement processes during attention switches, where the neural tracking of the new target stream emerges before disengaging from the previous target, revealing a transient simultaneous encoding of two speech streams. That transition was closely mirrored by a reduction in EEG alpha power, informing on the cognitive effort during different phases of the attention switch. We then isolated cortical activity reflecting lexical prediction mechanisms to determine how lexical context is updated after an attention switch, comparing four context-accumulation strategies that were constructed using Large Language Models. Our findings elucidate both the temporal and contextual mechanisms underlying auditory attention shifts, pointing to the possibility that listeners carry out a reset in lexical context after switching attention. By focusing on dynamic attentional reallocation, this study offers insights into the brain’s capacity for flexible speech processing in complex listening environments.
Citation: Carta S, Aličković E, Zaar J, López Valdés A, Di Liberto GM (2026) Competing speech streams are simultaneously represented in the human cortex during attention switching. PLoS Biol 24(7): e3003876.
https://doi.org/10.1371/journal.pbio.3003876
Academic Editor: Manuel S. Malmierca, Universidad de Salamanca, SPAIN
Received: July 3, 2025; Accepted: June 12, 2026; Published: July 16, 2026
Copyright: © 2026 Carta et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data supporting the findings reported in this manuscript are freely accessible without restriction. The EEG pre-processed dataset, the resulting analysis files, and the analysis code are publicly available on the open repository Zenodo (https://zenodo.org/records/20569817). The EEG recordings are provided following the Continuous-event Neural Data (CND) format standard. The associated speech stimuli can also be found in the same repository, within the STIMULI folder.
Funding: S.C., A.L.V., and G.D.L. were supported by the William Demant Fonden (https://www.williamdemantfonden.dk/), under grants 21 – 0628 and 22 – 0552, and by Taighde Éireann — Research Ireland (https://www.researchireland.ie/) under grant No. 18/CRT/6223. G.D.L. additionally conducted this research with the financial support of Research Ireland at ADAPT, the Research Ireland Centre for AI-Driven Digital Content Technology (https://www.adaptcentre.ie/) at Trinity College Dublin [grant 13/RC/2106_P2]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: EEG, electroencephalography; EOG, electro-oculography; EMG, electro-myography; ERSP, event-related spectral perturbation; iEEG, intra-cranial electroencephalography; FDR, false discovery rate; fMRI, functional magnetic resonance imaging; ICA, Independent Component Analysis; IQR, interquartile range; LLM, large language model; MEG, magnetoencephalography; PSD, power spectral density; RMS, root-mean-squared; SE, standard error; SEM, standard error of the mean; SNR, signal-to-noise ratio; SPL, sound pressure level; TRF, Temporal Response Functions
Introduction
To understand speech in multi-talker environments, listeners single out the target speaker from competing sound streams [1 – 3]. The neurophysiology of this selective attention process has been widely studied with simulated cocktail-party scenarios [4,5], shedding light on how our brains segregate a target stream from competing speech streams, and enabling the transformation of the target speech into linguistic meaning. While the extent to which masker speech streams are processed remains highly debated [6 – 8], there is no doubt that there are considerable differences between the processing of target and masker speech, which have been measured with various technologies, such as non-invasive electroencephalography (EEG) [1,9], intra-cranial electroencephalography (iEEG) [10], magnetoencephalography (MEG) [3,11] and functional magnetic resonance imaging (fMRI) [12,13]. That work could pinpoint precise loci in the auditory cortical areas where that segregation emerges [14] as well as measuring the substantial (but not total) suppression of linguistic processing for the masker speech [1,15 – 17]. However, neurophysiology literature in this field has almost entirely focused on sustained attention tasks [2,10], leaving considerable uncertainty on the neural underpinnings of attention switching.
Dynamic switching paradigms have been widely used in the domain of cognitive control studies to probe for cognitive flexibility and cognitive stability [18]. In those experiments, participants are often required to flexibly adapt their behavioral response depending on new instructions, initiating a task-switch [19 – 21]. For example, given a single digit, they are required to classify it either based on parity, i.e., whether it is even or odd, or based on relative magnitude, i.e., whether the digit is greater than or less than 5 [22]. In these paradigms, the switch-cost is the increase in reaction time or error rate when switching from one task to the other. Similar behavioral paradigms have also involved simple speech stimuli in multi-talker settings [23 – 25]. However, the main interest of those tightly controlled experiments was to model the process of target speech selection as one particular instance of a task-switching problem, i.e., target stream selection could either depend on spatial location or voice identity [23], rather than focusing on the dynamic aspect of attention re-allocation per se in naturalistic multi-talker scenarios. As such, very little is known on how a flexible reorienting of attention might impact speech processing of continuous competing streams.
In recent speech neurophysiology research, experimental paradigms have started to include switches of attention as a tool towards tailored EEG/MEG methodological advances in the domain of attention decoding [26,27], or to investigate how sustained speech attention unfolds for moving auditory objects [28]. However, to the best of our knowledge, only one previous study has specifically focused on the neurophysiology of attention switching in multi-talker scenarios, relating the neural encoding of speech during attentional re-orienting with EEG alpha activity and pupil dilation dynamics [29]. Those findings proved that the neurophysiology of attention switching can be studied non-invasively. Building on that work, our study sheds light on the exact neural dynamics supporting the steering of attention between two competing speech streams, disengaging from the previous target stream while engaging to the new one.
In this study, we measure the neural encoding of speech using a range of encoding window lengths, as listeners steer their attention from one speaker to another. We test whether engagement with a new speech stream begins before disengagement from the previous target is complete, resulting in a brief period of simultaneous tracking of both streams. Such an asymmetry in the disengagement-engagement processes, even if transient, could support the ability to explore alternative auditory streams while maintaining attention to a given stream [30].
The neural encoding of speech was measured from normal-hearing adult participants using EEG during an immersive multi-talker listening task. Participants were exposed to two competing speech streams from TED talks, presented via two front-facing loudspeakers, while background noise from a 16-talker speech babble played from rear loudspeakers (Fig 1A). An on-screen arrow cued participants to attend to one of the two speech streams and to shift their attention rapidly whenever the arrow changed direction, approximately every 10 – 30 s (Fig 1B). Neural tracking of target and masker speech was quantified using the Temporal Response Function (TRF), describing the linear relationship between each speech stream and the neural responses. As an initial validation, we confirmed that the attended stream could be reliably decoded from the EEG, consistent with the extensive literature on sustained attention [9,10,31]. This confirms that the EEG responses in this experiment reflects differential encoding of target versus masker speech (Fig 1C).
Fig 1. Experiment overview and validation.
(A) Participants were presented with speech from two loudspeakers placed in front of them with 60° of separation (30° left and 30° right), and with concurrent 16-talker background noise (B1–B4). In each trial, the screen presented an arrow pointing to the target speech stream. Participants were instructed to switch attention as soon as the visual cue changes direction. (B) Schematic diagram of one experimental trial. The black area represents blocks of attention either to the left (L) or right (R) front streams. The red arrows indicate the instants where the attention cue switches side (six times per trial). Note that block duration was randomized and always between 15 and 30 s, with trials lasting 3 min. (C) EEG data validation was carried out by running an attention decoding analysis. Progressively longer decoding windows were considered (larger windows use more data, typically leading to more accurate decoding scores). Binary classification scores are reported arbitrating between the target and masker streams. The dashed line indicates the 95th percentile of a random distribution calculated by randomizing the classification labels. Statistically significant attention decoding classification scores were measured for all the decoding windows considered, with numerical results comparable with previous studies on selective attention [31,34,35]. Data supporting this figure is available at: https://zenodo.org/records/20569817.
https://doi.org/10.1371/journal.pbio.3003876.g001
We next addressed two fundamental questions about the neural mechanisms underlying attention switching in naturalistic listening. First, we asked whether the processes of engaging with a new speech stream and disengaging from a previous one unfold symmetrically (Figs 2 and 3). To test this, we fit encoding TRF models to EEG data, measuring the neural tracking of the two competing speech streams over time. This allowed us to characterize the average encoding dynamics surrounding attention switches, comparing disengagement and engagement processes. The second objective was to understand how our brains update and use lexical context when switching attention (Fig 4). Building on previous work showing that speech comprehension is supported by contextual predictions [32,33], we formulated four competing hypotheses reflecting different assumptions about how linguistic context is preserved, reset, or selectively updated across an attention switch. Using a state-of-the-art large language model (LLM), we derived quantitative predictions for each hypothesis, resulting in four regressors for lexical surprisal and entropy, separately, differing in their sensitivity to prior context and to the occurrence of the switch. Encoding TRF models were then fit for each hypothesis, allowing us to compare alternative context-accumulation strategies and identify the model most consistent with the observed neural responses. This study provides substantial new insights into the temporal unfolding and contextual mechanisms guiding attention switching, encompassing both low and high levels of speech abstraction.
Fig 2. The attention-switching cue prompts a robust disengagement from Speaker 1 and engagement to Speaker 2, and it is followed by a significant decrease in the EEG alpha ERSP.
Disengagement has longer temporal dynamics compared to engagement. (A) Left: Speech tracking encoding for an attention switch from Speaker 1 and 2. The trajectory in the panel represents our null hypothesis, where the disengagement and engagement processes progress in a symmetric manner after the switch-cue (vertical gray line). Right: Results for the neural tracking of Speaker 1 and Speaker 2 across the switching cue. EEG prediction correlations (average across all channels) obtained from a 4-s sliding-window TRF model including Envelope (Env), Word Onset (WO) and Word Surprisal (WS) features. Coloured horizontal bars at the bottom of the plot indicate the attention instruction around the attention switching cue. The turquoise dot indicates the encoding switch of EEG prediction correlations based on Spk1- and Spk2- speech features. The piecewise linear model fit for disengagement and engagement is overlayed on the EEG prediction correlation values. Please note that the broken-line-fit in this plot was performed on the grand-average cortical tracking curves here for illustrative purposes. Please find the estimates at the single-participant level in Panel C. Hexagram shapes indicate the start of the disengagement (blue) and engagement (yellow) processes, while diamonds represent the end of the transitions. (B) Left: Diagram of expected results for alpha-band ERSP (event-related spectral perturbation) across the switching cue. Right: ERSP of the alpha band (8 – 12 Hz) around the switching cue (average of all channels), computed with a 4-s sliding window, as above. Scalp topographies at selected time points reveal a pattern of posterior negativity, which drops significantly following the instruction to switch (thick black lines indicate a statistically significant change compared to pre-switch baseline). The red dot represents the average of ERSP minima across participants. The shaded area represents the standard error of the mean (SEM) across participants. (C) Left: Comparison of encoding switch of EEG prediction correlations (turquoise bar) and alpha ERSP minimum (red bar) for a 4-s sliding window. The alpha ERSP reaches its minimum significantly after the Spk1-Spk2 encoding switch point. Right: Comparison of temporal dynamics for start and end points of disengagement and engagement processes, with start/end transition points estimated at the single-participant level. Stars indicate significant statistical effects (paired sample t-tests; *p ≤ 0.05; **p ≤ 0.01; **p ≤ 0.001). Data supporting this figure is available at: https://zenodo.org/records/20569817.
https://doi.org/10.1371/journal.pbio.3003876.g002
Fig 3. Comparing the start and end transition points for the disengagement and engagement processes after the attentions switching cue.
The process of engaging to a new speaker begins and ends significantly earlier than disengaging from the previously attended speaker. (A, B) Start and end points of the transition for the disengagement (blue) and engagement (yellow) processes over five TRF sliding window lengths. Error bars represent SEM across participants. Stars indicate significant effects of process type (two-way repeated measures ANOVA; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001). Data supporting this figure is available at: https://zenodo.org/records/20569817.
https://doi.org/10.1371/journal.pbio.3003876.g003
Fig 4. Investigating lexical prediction mechanisms during attention switching.
(A) Layout of the four context models. Blocks coloured in black illustrate sustained attention either to the Left or Right stream, while orange arrows indicate attention switching cues. The thick red arrow indicates the context used to guide word predictions for the current block (B7, highlighted in orange). (B) Average lexical entropy at words preceding and following the attention switch cue. Note that no value for entropy is displayed in the Reset model for the first word after the switch, due to the context being fully reinstated. (C) EEG prediction correlations for the four multivariate TRF models, only differing in their entropy feature. Coloured dots indicate the average across all electrodes and participants. The gray area at the bottom represents the average encoding accuracy of a multivariate TRF without any semantic information (Envelope + Word Onset). Stars represent statistically significantly greater EEG prediction correlations for the Reset model compared to the other models (Significance levels: *p < 0.05, **p < 0.01, ***p < 0.001). Topographical patterns illustrate the gain due to semantic information (compared to the Envelope + Word Onset TRF) for the four models. (D) (Left) TRF weights for the entropy feature at time-lags between −100 and 600 ms relative to stimulus onset. Transparent shaded areas represent the standard error of the mean (SEM) across participants. The horizontal black line indicates the time window employed to compute the average TRF-N400 amplitude. (Right) Boxplots representing the distribution of the TRF-N400 amplitude across participants for the four context models. The central line within each box represents the median, while the edges of the box indicate the interquartile range (IQR). Whiskers extend to the most extreme data points within 1.5 times the IQR from the quartiles. Outliers are plotted as individual points beyond the whiskers. Stars indicate statistically significant differences (Significance levels: *p < 0.05, **p < 0.01, ***p < 0.001). Data supporting this figure is available at: https://zenodo.org/records/20569817.
https://doi.org/10.1371/journal.pbio.3003876.g004
Results
Behavioral performance
Following each trial, participants were first presented with a four-alternative forced-choice question about the content of the attended speech stream to confirm task engagement. Behavioral performance revealed that they were able to successfully reply to content-related questions, with an average accuracy of 86.3% (SEM 2.6%). Participants were also required to indicate their preference between left and right streams, which was found to be overall balanced, with the left stream selected in 49.79% of the trials, on average (SEM 1.7%). Finally, perceived difficulty of the attention switch for every trial was measured by asking participants to rate it on a scale from 1 (very easy) to 5 (very hard). The average difficulty of the switch was judged to be 3.1 out of 5, with a SEM across participants of 0.11 points. Due to technical issues, behavioral data for one of the 24 participants was not available, therefore behavioral performance was computed based on the data from the remaining 23 participants.
Decoding of selective auditory attention in a dynamic switching scenario
Participants’ attention was decoded with a backward TRF analysis, describing the relationship between the EEG signals and the envelope of the target speech. For each left-out trial, the speech envelope reconstructed from the target decoding model was correlated with the envelopes of both the left and right speech streams. Attention was classified by determining which speech stream’s envelope showed a higher correlation with the reconstructed envelope. Since this was a dynamic attention-switching scenario, the attended speech could alternatively correspond to the left or the right stream. Classification was considered correct when the reconstructed envelope correlated more strongly with the target speech envelope than with the masker envelope. Classification accuracy was then computed as the proportion of instances where this criterion was met. To establish chance performance, left and right labels were randomly shuffled 100 times for each decoding window. As shown in Fig 1C, the longer decoding windows led to higher classification performances. However, even with a 1-second window, classification accuracy was significantly above chance level, and all decoding windows yielded classification rates significantly above the 95th percentile of its chance distribution (paired two-tailed t test, FDR-corrected for multiple comparisons for windows of 1 s, 2 s, 4 s, 8 s, 16 s, 32 s, respectively: p = 0.47e−9; 0.53e−9; 0.27e−9; 0.24e−9; 0.24e−9; 0.24e−9). These findings align with previous work decoding sustained attention or employing match-vs-mismatch classification metrics [9,31,36], and confirm that a classification based on the envelope reconstruction can reliably track selective attention even during attention switches.
Neural tracking of competing speech streams in a dynamic switching scenario reflects the listener’s focus of attention and is related to changes in alpha ERSP
A multivariate TRF analysis was carried out to characterize the neural tracking of two competing speech streams in a setting where participants were instructed to dynamically switch their attention between the two streams. Single-subject TRFs were trained on the target stream and tested on both speakers (i.e., Spk1 and Spk2) using a multivariate speech representation that included Envelope, Word Onset and Word Surprisal features (for more details see Methods). EEG prediction correlations were computed using a sliding window to correlate true and predicted EEG signals over time, with a leave-one-out cross-validation procedure and were averaged across all EEG channels. Importantly, because these correlations are computed using sliding windows, the resulting switch timing depends on the sliding window length. As such, the temporal dynamics deriving from our analyses do not reflect the exact timing of the underlying neural processes, and they should always be interpreted with the caveat of the sliding window length.
In order to analyze robust switching dynamics, we selected 21 participants displaying a reliable attentional bias over the course of the switch, based on an above-chance classification accuracy criterion (>50%) over the course of the switch. In doing so, we removed participants for whom the start and end points of the (dis)engagement could not be estimated (note that this exclusion is determined before identifying the start/end estimates; in that sense, this is different from an outlier removal, which would exclude extreme start/end values instead).
Aligning with our expectation (Fig 2A), EEG prediction correlations around the switching cue reflected tracking of Spk1 and Spk2 streams consistent with the attention instructions, such that Spk1 was significantly more tracked than Spk2 before the switch, while the reverse pattern was observed after the switch (paired two-tailed t test of Spk1-Spk2 difference against zero, FDR-corrected for multiple comparisons, p < 0.005).
As the attention switch unfolds, also the grand-mean ERSP in the alpha frequency band displayed a statistically significant change compared to baseline (one-sample t test against zero, FDR-corrected for multiple comparisons), revealing a pattern of occipito-parietal negativity in the scalp topographies (Fig 2B). This is consistent with our expectation of an impact of attentional reorientation on the EEG alpha band, which has already been shown to reflect attention switching behavior in competing speech listening scenarios [29].
EEG prediction correlations for Spk1 and Spk2 converged, before significantly separating again once the switching process was concluded and, presumably, the attention was fully reallocated. Here, we refer to the time point when EEG prediction overlaps between Spk1 and Spk2 as the encoding switch point. Given the observed statistically significant drop in alpha ERSP, we asked how the temporal dynamics of this drop compared to those of the EEG prediction correlations. To address this, for each participant, we identified the time of the alpha ERSP minimum, and the encoding switch point, based on an encoding window of 4s (Fig 2C). The choice of this particular encoding window for our main analysis is justified based on the classification accuracy results (Fig 1C), since it is a good compromise between temporal resolution and classification performance. However, the same pattern of results holds when considering multiple encoding windows simultaneously (S1 Fig). A paired t test comparing the temporal dynamics of the alpha ERSP and the EEG prediction correlations showed that the minimum of the alpha ERSP drop significantly follows the encoding switch point (t(20) = 4.29, p = 3.59e-4, Cohen’s d = 0.94).
We then evaluated multiple encoding window lengths, assessing the effect of Metric (encoding switch versus ERSP minimum) and Window (1, 2, 4, 8 s) on the timing of the encoding switch and the minimum of the alpha ERSP with a 2-way repeated measures ANOVA. The analysis revealed that the temporal dynamics of both encoding switch and alpha ERSP minimum became longer as the encoding window length increased (F(1.68, 33.55) = 52.77, p = 2.3e−10, ηp2 = 0.72; a Greenhouse-Geisser’s correction applied due to sphericity violation), which is unsurprising given the methodological constraints we discussed (see Methods). More interestingly for our question, a statistically significant effect of Metric emerged (F(1,20) = 20.26, p = 2.18e−4, ηp2 = 0.5), with the alpha ERSP minimum occurring significantly later than the encoding switch across a range of encoding windows (Holm-corrected post-hoc t test: t(20) = 4.5, p = 2.18e−4, Cohen’s d = 0.97).
Dissecting the temporal dynamics of attentional disengagement and engagement during attention switching
The attention switching cue prompts the listener to reallocate their attention from the previously attended speaker, Spk1, to the newly attended speaker, Spk2. While this re-routing of attention appears to be a single, unified process, it is possible to distinguish two separate operations that are necessary for it to happen: disengagement, which we define as the decrease in neural tracking for the previously attended speech stream, and engagement, which we define as the increase in neural tracking for the previously unattended speech stream. Our goal was to clarify the temporal dynamics of these two operations to understand whether they occur fully in parallel, serially, or with a certain degree of overlap. It is worth noting that, due to the use of sliding encoding windows, the estimated temporal dynamics of engagement and disengagement do not reflect the exact time course of the underlying neural processes and should be interpreted as relative, rather than absolute, temporal metrics. As in the previous analysis, we first selected participants displaying a reliable attentional bias over the course of the switch (see Methods). For this selection of 21 participants, we fitted a piecewise linear regression on single-subject EEG prediction correlations, and found the optimal breakpoints, corresponding to the start and end time points of disengagement and engagement (Fig 2C). As above, we chose to focus on an example window of 4 s and later replicated our results on a range of encoding window lengths. To further characterize the spatial patterns of engagement and disengagement processes, scalp topographies of the EEG prediction correlations at selected time points are shown in S2 Fig, indicating that the most predictive channels were predominantly located over central-parietal regions. Disengagement and engagement processes were compared separately based on their start times and end times, revealing consistently earlier temporal dynamics for the engagement compared to the disengagement. Engagement to the newly attended speaker started significantly earlier than the disengagement from the previously attended speaker (paired-sample t test: t(20) = 2.37, p = 0.03, Cohen’s d = 0.52), and finished significantly earlier (paired-sample t test: t(20) = 2.35, p = 0.03, Cohen’s d = 0.39).
We then extended our analysis to a range of sliding window lengths and compared start times and end times for disengagement and engagement processes, including Window (1, 2, 4, 8, 16 s) and Process (disengagement versus engagement) as main factors in a repeated measures ANOVA. Regarding the start points (Fig 3A), our analyses revealed an expected statistically significant effect of Window (F(1.51,30.14) = 9.7, p = 0.001, ηp2 = 0.33; the assumption of sphericity was not met; hence, a Greenhouse–Geisser’s correction was applied), with longer temporal dynamics corresponding to longer encoding window lengths. More importantly, we also observed a significant main effect of Process (F(1,20) = 5.48, p = 0.03, ηp2 = 0.21), with engagement to the newly attended stream starting significantly earlier than the disengagement to the previously attended stream (Holm-corrected post-hoc t test: t(20) = 2.34, p = 0.03, Cohen’s d = 0.54). The same statistical analysis was repeated separately on the end time points of disengagement and engagement processes (Fig 3B), revealing once again a main effect of Window, whereby longer encoding windows yield longer temporal transitions (F(1.97,39.32) = 31.76, p = 7.2e−9, ηp2 = 0.61; the assumption of sphericity was not met; hence, a Greenhouse–Geisser’s correction was applied). A significant main effect of Process also emerged (F(1,20) = 4.46, p = 0.047, ηp2 = 0.18), revealing that the process of engagement to the newly attended speaker, not only starts, but also ends significantly earlier than the disengagement (Holm-corrected post-hoc t test: t(20) = 2.11, p = 0.047, Cohen’s d = 0.58).
A follow-up analysis including the three participants with lower-than-chance classification accuracy around the switching cue confirmed that these data points introduced noise to the estimation of engagement and disengagement latencies. This was expected, as the start and end transition points cannot be determined in those participants. The patterns observed were qualitatively similar to the main result reported above, with earlier temporal dynamics for the engagement compared to the disengagement, albeit with weaker effects below the statistical significance threshold (repeated-measures ANOVA; start point: F(1,23) = 2.69, p = 0.11, ηp2 = 0.1; end point: F(1,23) = 2.96, p = 0.1, ηp2 = 0.11).
Determining how lexical predictions are built during attention switching
Reorienting attention to a different speech stream implies a change of context and, consequently, different semantic priors for lexical predictions. We thus hypothesized that incorporating this change of context into the structure of our semantic regressor in a multivariate encoding TRF model would increase EEG prediction correlations, as it would better reflect the dynamically updating neural tracking of the competing speech streams. We compared four alternative models representing how context could be incrementally accumulated for performing lexical predictions at one particular attention block (e.g., B7, in Fig 4A). A naïve Oracle model, which uses all available context of previous blocks from the current stream, whether attended or unattended, to predict words from the current block, served as our baseline, since it was essentially a switch-unaware contextual representation. Speaker-Specific and Attention models were instead switch-aware models, as they only considered previously attended blocks as part of the context for lexical predictions. Speaker-Specific assumed a higher degree of stream segregation, since its context only consisted of previously attended blocks from the same speech stream, while Attention included any previously attended block from both streams. The Reset model instead ignored all previously attended blocks from any of the streams and computed context only over the course of the current block of attention, as if the priors for lexical predictions were reset at each attention switch (Fig 4A).
As lexical entropy is a proxy of uncertainty for next-word prediction, its values should be impacted by a switching cue, which determines an abrupt change of context. Fig 4B shows the change of average lexical entropy values in words preceding and following the switch cue, which vary depending on the context models. It can be observed that the Reset model peaks with the highest uncertainty and slowly decays over the course of the next words, while the Attention and Speaker Specific models have overall similar lexical entropy dynamics and more stable values. Consistently with its switch-unaware nature, the Oracle model instead displays entropy values that are largely unchanged despite the switch. An explicit comparison of the average entropy values of the four context-accumulation strategies revealed statistically significant differences (repeated-measures ANOVA, Greenhouse-Geisser correction due to sphericity violation; F(1,19) = 39.57, p = 9.59e−10, ηp2 = 0.68). Post-hoc pairwise tests (Holm-adjusted) indicated that the Reset model showed an intermediate average entropy, significantly higher than the Oracle model (t(19) = 5.75, p = 1.44e−6, Cohen’s d = 0.45), and significantly lower than the Attention (t(19) = 4.64, p = 6.28e − 5, Cohen’s d = 0.36) and Speaker-Specific (t(19) = 2.24, p = 0.04, Cohen’s d = 0.17) models. As such, despite showing the highest peak following the attention switching cue, the Reset model had overall intermediate entropy values across the four lexical expectation models considered here.
Lexical surprisal and lexical entropy were used as semantic information regressors for each context model and separately included in a multivariate stimulus representation to fit single-subject encoding TRFs (Envelope-Word Onset-Word Surprisal and Envelope-Word Onset-Word Entropy). Resulting TRF weights and EEG prediction correlations were then compared across context models, with the hypothesis that switch-aware and context-rich representations (e.g., Speaker-Specific or Attention) would best describe neural activity in attention-switching scenarios.
Before comparing the context models, we first tested whether each of them yielded a significant encoding accuracy gain compared to the baseline model only consisting of acoustic features (Envelope and Word Onset). When using entropy as a regressor for semantics, all models, with the exception of Oracle, showed a statistically significant gain, suggesting a robust tracking of semantic information in addition to the stimulus acoustics (paired t-tests: Oracle versus Acoustics: p = 0.2; Spk.Spec. versus Acoustics: p = 0.04; Attention versus Acoustics: p = 0.04; Reset versus Acoustics: p = 0.002). Employing word surprisal as semantic regressor yielded similar results, with all the models showing a robust encoding of semantic information, apart from Oracle (paired t-tests: Oracle versus Acoustics: p = 0.15; Spk.Spec. versus Acoustics: p = 0.02; Attention versus Acoustics: p = 0.02; Reset versus Acoustics: p = 0.01). The non-significant gain of the Oracle model compared to the acoustic model was expected, since Oracle was designed as a control switch-unaware model.
In contrast to our expectation, the Reset context model was shown to yield higher EEG prediction correlation values when entropy was used as a regressor for semantics (Fig 4C). A repeated measures ANOVA revealed a statistically significant effect of the main factor, Context Model (F(2.1,47.75) = 9, p = 4e−4, ηp2 = 0.28; with Greenhouse-Geisser’s correction). In the Holm-corrected post-hoc tests, the Reset model was shown to yield significantly higher encoding accuracies than Oracle (t(23) = 4.99, p = 2.63e−5, Cohen’s d = 0.14), Speaker Specific (t(23) = 3.73, p = 0.002, Cohen’s d = 0.1), and Attention (t(23) = 3.28, p = 0.006, Cohen’s d = 0.09). We then assessed the difference of TRF weights for the entropy feature across the four context models (Fig 4D), averaging the weights’ amplitude within a window broadly centered around the TRF-N400 latency (350 – 550 ms). A repeated measure ANOVA was run on the weights’ amplitude values, revealing a main effect of Context Model (F(3,69) = 15.51, p = 8.2e−8, ηp2 = 0.4). Post-hoc tests (Holm-corrected) showed that weights for the Reset model had lower TRF-N400 amplitude compared to Oracle (t(23) = −5.56, p = 2.4e−6, Cohen’s d = 0.45), Attention (t(23) = −5.84, p = 9.2e−7, Cohen’s d = 0.47), and Speaker Specific (t(23) = −5.24, p = 6.5e−6, Cohen’s d = 0.43).
When fitting a multivariate TRF including lexical surprisal as a semantic regressor, we observed a statistically significant difference in EEG prediction correlations between the four context models (F(1.59,36.6) = 3.96, p = 0.04, ηp2 = 0.15, with Greenhouse–Geisser correction). Post-hoc analyses indicated a statistically significant difference between the Reset and Oracle models (t(23) = 3.18, p = 0.013, Cohen’s d = 0.1), while all other post-hoc pairwise comparisons did not reach the significance threshold (p < 0.05). Similarly, no statistically significant difference emerged when comparing the TRF-N400 amplitude of the models’ TRF weights.
Discussion
Speech communication in multi-talker environments requires a skillful combination of sustained attention and rapid attention switching abilities [5,30]. While the neurophysiology of sustained speech attention has been widely studied [1,9,37,38], less is known about the neural mechanisms of attention switching. Here, we fill this gap with a tailored EEG experiment examining the neurophysiology of attention switching across different levels of speech abstraction. In doing so, we (1) demonstrated an experimental paradigm that can successfully probe both sustained attention and attention switching mechanisms; (2) successfully dissected disengagement and engagement processes with a high temporal resolution, identifying substantial asymmetries in their temporal unfolding and a transient simultaneous encoding of two speech streams; and (3) proposed a neurophysiologically plausible explanation of how our brains update and use lexical context when switching attention.
The findings in this study have several implications for our understanding of speech attention switching mechanisms. The asymmetry measured between disengagement and engagement processes highlights the importance of studying the two processes separately. That distinction was often not considered in previous studies on sustained attention, which often focused on measures of attention bias or classification [10,31,34,39]. The effectiveness of such decoding metrics has been a driving force for research on brain-computer interfaces such as cognitively-controlled hearing devices [40 – 43]. Our finding highlights that encoding metrics enable a sufficient level of detail for disentangling how the encoding of different streams evolves over time. Here, we measured an asymmetry between disengagement and engagement processes during attention switching in a very specific scenario. Indeed, it will be important to determine how that relationship is modulated by factors such as cognitive load, aging, cognitive abilities, hearing difficulties, interest in the speech content, frequency of attention switches in a trial, among many others. Of course, future work should also scrutinize how the specific nature of the task might impact that phenomenon.
Sustained attention tasks, where participants focus on a target speech while ignoring the masker [2,44], involve a quite particular scenario where listeners have no incentive to monitor unattended streams. In real-life situations, however, listeners may have reasons to explore alternative speech streams, for example, due to a lack of interest in the current speaker. Our experimental paradigm more closely mirrors this scenario. Although the instructed nature of the task makes it less realistic, the paradigm incentivises monitoring the masker and being ready to rapidly switch attention, which contrasts with sustained attention tasks. While the asymmetry observed may be specific to this experimental paradigm, the result indicates that our brains can engage with a new target even before starting the disengagement from the previous one, leading to a transient simultaneous tracking of the two streams, compatible with auditory scene monitoring mechanisms [45,46]. In other words, there is a brief period, following an attention switch, where the tracking of a new stream begins to emerge without altering the tracking of the previous stream. The engagement and disengagement latencies are window-dependent and, as such, are not intended as absolute neural timings. Nonetheless, the engagement-disengagement asymmetry is robust to the selection of the sliding-window length (Fig 3) and cannot be explained by the temporal smoothing introduced by the window or by trial- and participant-averaging. The temporal smoothing could, in principle, temporally stretch the engagement-disengagement dynamics, but not generate an asymmetry. Future research could investigate the variability across trials and participants to better characterize these temporal dynamics. Within the dorsal–ventral attention framework [47,48], attention reorienting results from the flexible interplay of the goal-directed dorsal network and the stimulus-driven ventral network. Our finding of an engagement-disengagement asymmetry aligns with this view of an integrated process of attention release and reallocation, with potentially overlapping neural dynamics.
Intuitively, maintaining a transient parallel representation of multiple speech sources during attention switching is an efficient neural processing strategy. It allows the flexibility to switch back to the previous stream, if necessary, without fully committing to the newly attended speech immediately. This phenomenon supports previous claims that our brains can process speech maskers beyond the acoustic level, encoding linguistic properties to some extent [7,8,16]. Unattended speech streams are also represented in human cortical activity, with evidence for lower encoding strengths or longer time latencies than the target stream [1,49,50], and with a gradient of attentional bias from primary to nonprimary auditory cortex [13,14], whereby the unattended stream encoding tends to be substantially reduced or not measurable in higher-order cortical areas [51,52]. Prior research has shown that not only the speech envelope, but also other key features of the unattended speech, such as acoustic onsets, are neurally represented [53,54] and, when not readily available due to speech masking, they are even restored at later temporal scales [55]. One interpretation is that encoding a template structure of the unattended speech might be a useful strategy to suppress it [53]. Other research found attentional fluctuations corresponding to the changes in Target-Masker relative sound energy, with evidence that our brains may encode some phonetic information of unattended streams [56,57]. The encoding of such unattended stream information may be one of the factors facilitating the rapid engagement during attention switching.
Using alpha ERSP as indicator of listening effort, this study related perceptual demands with attention switching dynamics. A large reduction in EEG alpha-band power was measured consistently about 4.5 s after the attention-switching cue. The ERSP trajectory suggests that a strong listening effort persists throughout the attention switch, with a substantial reduction near the end of the switch (Fig 2A). Interestingly, the trough of the alpha ERSP observed in this study roughly corresponds to the moment when the new target stream becomes fully tracked, i.e., when the engagement process is completed, which is well before completion of the disengagement process. Since the attention switching process can be deemed completed when cortical tracking measurements return to pre-switch levels for both streams, this result points to a link between alpha power and the engagement process specifically. Another possibility is that the alpha ERSP dynamics reflect a combination of listening effort while refocusing attention on the new target stream and active suppression of the new masker stream. When the newly attended stream is tracked at pre-switch levels, a sufficient acoustic and linguistic context on that stream may have been accumulated to facilitate the tracking, releasing cognitive resources. Future studies could explore this possibility by examining how the switch difficulty influences the correspondence between cortical tracking asymmetry and alpha ERSP. This finding extends prior research on the neural correlates of selective attention and listening effort. Variations in alpha-band activity have been associated not only with auditory attention effort [58 – 62], but also with variations of internally- versus externally-focused brain states [63], and with the active suppression of irrelevant information in accordance with behavioral goals [64 – 66]. Notably, cognitive load and the inhibition of irrelevant stimuli appear to be even more strongly influenced by attention reorienting than by maintaining attention on a single speaker [29,67], highlighting the increased cognitive demands of switching attention. While alpha power has commonly been linked to subjective listening fatigue [58,60] or the signal-to-noise ratio (SNR) between attended and unattended streams [61], it is less frequently associated with neural markers of speech tracking [68,69]. In this study, we identified a link between the temporal dynamics of neural speech tracking and listening effort, suggesting a potentially valuable metric for future research into attention-switching challenges.
While cortical tracking and alpha ERSP metrics can be used to investigate how attention is dynamically reallocated between competing streams, they do not specify how higher-level linguistic representations are updated during the attention switch. Considering the importance of context in speech comprehension, it is particularly relevant to investigate how it is adjusted while shifting attention from one speech stream to another. Addressing this question requires focusing on neural representations with relatively extended temporal dynamics, which can be meaningfully compared across competing models, and with sufficient distinction from the speech envelope tracking to enable the isolation of related neural signatures. Lexical entropy and surprisal, with their long-latency and widely studied neural signatures, are an ideal choice for characterizing the impact of context on attention switching. Other informative speech properties such as phonology or prosody would also be important to examine; however, their fast temporal dynamics challenge the isolation of their attention-switching dynamics from the speech envelope tracking. We therefore focused on how semantic predictions are updated following an attention switch, modeling whether linguistic context is maintained, reset, or updated during the process.
This study compared four context-accumulation models that varied in their sensitivity to attention switches and access to prior context: an Oracle model (context-rich but switch-unaware), Speaker Specific and Attention models (both context-rich and switch-aware but differing in stream selectivity), and a Reset model (aware of the switch but limited to the current block’s context). For each model, a multivariate TRF was fit using a semantic regressor aligned with the respective context strategy. Our data indicates that the Reset model best predicted EEG data, outperforming models that retained past context and challenging the assumption that prior semantic information aids comprehension during attention switches. This finding was unexpected but one of the possible outcomes that we had hypothesized, and it may suggest that listeners reset context and recalibrate their lexical predictions dynamically when switching attention to a new stream, in line with findings in the episodic memory and event-segmentation literature [70 – 72].
Interestingly, the multivariate TRF analysis revealed fine-grained differences among the four context models when lexical entropy is used as the semantic regressor, but not for lexical surprisal. This may reflect the forward-looking nature of entropy, as opposed to the reactive nature of lexical surprisal. Another consideration is that participants actively performed the attention switch, therefore expecting to encounter different speech material, potentially dampening the surprisal. This was not the case for the LLM model, which did not receive a switch cue. This possible mismatch between LLM and human brain could impact surprisal but not entropy (as that reflects the process of context-building), providing a potential explanation of why the Reset model performs best only for the entropy regressor. For interpreting these results, it is also important to consider that LLMs like Mistral are optimized for next-word prediction, without a requirement of being neurophysiologically plausible. While one interpretation is that our brain’s lexical predictions are built after resetting the context, it is also possible that Mistral LLM and our brains deal with these speech discontinuities in a different way altogether. With these caveats in mind, counter to consistent reports of the similarity between modern LLMs with neurophysiological activity [73 – 75], the higher neurophysiological plausibility of the Reset model suggests that prior context is not availed of by our brains in the way implemented by the competing models, Attention and Spk-Specific.
Other strategies making use of the prior context might also be in place. One possibility is that switching attention prompts a different use of context, for example summarizing its abstract meaning, as the gist of the story [16]. As such, there could be value in exploring different strategies for context representation, for example, by employing Large Concept Models [76], which are trained and optimized for sentence prediction. This latter possibility is also supported by recent work on the accumulation of linguistic context in LLMs and the human brain [77] while listening to continuous monologues, showing that LLMs with a limited context window (32 tokens) and with access to a coarse summary of the previous context predict neural activity better than LLMs with a higher token-memory.
In summary, this study showed that the process of attention switching in a realistic multi-talker scenario can be investigated in terms of its engagement and disengagement components, with a transient parallel representation of the two streams. We highlight the importance of relating metrics of neural tracking of speech with metrics of listening effort and demonstrate that the listening effort starts decreasing following successful disambiguation of the two streams during attentional re-allocation. Finally, we introduce an approach for modeling lexical context of dynamic attention scenarios, showing the sensitivity of transformer-based language models to subtle differences in context accumulation strategies. These findings have implications for future investigations into the cortical mechanisms of attention re-orienting and can be employed to highlight differences across diverse populations in terms of age and hearing levels.
Methods
Ethics statement
Written informed consent was obtained from the study participants. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the School of Psychology Research Ethics Committee of Trinity College Dublin (ethics approval number: SPREC012023 – 08).
Participants and experimental procedure
We recruited 24 young native English speakers (between 18 and 39 years of age) to take part in the study. Participants had normal hearing, as per a screening pure tone audiogram from 0.25 Hz to 8 kHz and reported no history of neurological or psychiatric disorders and had normal or corrected-to-normal vision.
The experiment simulated a multi-talker scenario (Fig 1A) with a circular array (1.50m radius) of six loudspeakers surrounding the listener (at horizontal angles of ±30°, ±112.5°, and ±157.5° relative to the participant). Participants were instructed to dynamically switch their attention between left and right speech streams in the foreground, following a visual cue (left- or right-pointing arrow) indicating the to-be-attended side, which was displayed at the center of a screen placed in front of them. While switching their attention between the foreground streams, they were also asked to ignore a 16-talker noise played from the four loudspeakers in the background (B1-B4, each of them delivering a 4-talker babble). Frontal streams were presented at 60 dB sound pressure level (SPL) each, while each of the noise babbles was delivered at a level of 54dB SPL, resulting in a 3dB SNR of the foreground relative to the background.
Participants were presented with 20 trials (lasting 180 s each) and had to perform 6 attention switches per trial, occurring at semi-random intervals (Fig 1B). For this reason, blocks of sustained attention to one particular speech stream varied considerably in duration, spanning from 10 to 30 s. For each trial, a different male and female speech stream was played from the left and right loudspeakers in the foreground, counterbalancing throughout the experiment for side of presentation and start of the attention block (i.e., the experiment consisted of five sub-blocks with the following trial sequence: Male Left — Attention Start: Left; Female Left — Attention Start: Left; Male Left — Attention Start: Right; Female Left — Attention Start: Right).
Each trial started with the visual cue pointing towards the to-be-attended side and background noise only, followed by the two foreground speech streams starting simultaneously after 5 s. At the end of each trial, participants answered three multiple-choice questions. First, they were presented with a four-alternative forced-choice question regarding the content of the attended speech stream. As attention alternated between the left and right speech streams over the course of the trial, the question could be about either of the two competing streams. The second question was a binary choice assessing participants’ preference between the two streams (left or right) based on personal interest. Finally, a 5-point Likert-scale question (1: very easy; 5: very hard) was used to quantify the perceived difficulty of the attention switching task. The experiment flow was self-paced and, to minimize fatigue, it included three mandatory breaks, each lasting no less than five minutes, every fifth trial.
Speech streams were presented at a sampling rate of 44.1 kHz, delivered through a Roland Octa-Capture 10 × 10 sound card (24-bit/192 kHz), and played through six PreSonus Eris 4.5BT loudspeakers. Participants’ EEG activity was recorded using a BioSemi ActiveTwo system at a sampling rate of 512 Hz, from 64 electrodes positioned on a standard cap following the International 10/20 system. An active (CMS) and a passive (DRL) electrode were used as reference for all electrodes, and two additional electrodes were placed on the mastoids for offline referencing. For 21 of our 24 participants, we additionally recorded electro-oculography (EOG) and electro-myography (EMG). Two electrodes were placed on the left and right temples to capture horizontal eye movements, and two electrodes were positioned above and below the left eye to record vertical eye movements and blinks. To capture EMG activity related to head rotation, an electrode was placed on the left deltoid muscle. Please note that activity from these external electrodes has not been analyzed as part of this study.
Stimuli
The foreground speech stimuli included 40 TED Talks covering a range of topics, with 20 female and 20 male presenters, each speaking in a variety of English accents. All speech streams were root-mean-squared (RMS) normalized to reduce differences between male and female voices. Each of the 4-talker background babble signals was obtained by summing the audio signals of four separate TED talks. The long-term average spectrum of the babble noise was then adjusted to align with the overall spectrum of both male and female foreground speakers, to prevent inconsistencies in masking.
EEG data preprocessing
Neural data were analyzed with custom scripts in MATLAB software (MathWorks), based on publicly available scripts and resources shared as part of the CNSP initiative (Cognition and Natural Sensory Processing; https://cnspworkshop.net). Neural signals were first band-pass filtered between 0.5 Hz and 8 Hz, using a zero-phase shift Butterworth filters of order 4, and then downsampled from 512 to 64 Hz. Spherical spline interpolation was applied to replace channels that were three standard deviations away from the mean. EEG was then re-referenced to the average of the two mastoid channels.
Speech features
The current study aimed to characterize the neural tracking of a dynamic multi-talker scenario by measuring the relationship between EEG data and various features of the foreground speech stimuli, related to their acoustic and lexical properties. To model the speech acoustics, the audios’ broadband amplitude envelopes were extracted by taking the absolute of the Hilbert transform. In order to model the lexical properties of speech, the transcribed stimuli and their audios were first automatically aligned using the WebMAUS Basic aligner [78 – 80], which identified timestamps corresponding to the start and end of each word. The resulting automatic alignment was saved in the TextGrid format and adjusted manually, when necessary, using the Praat software [81]. The time stamps were then used to build binary word onset vectors in MATLAB. While binary word onset vectors represent information related to word segmentation, they can also be modulated according to each word’s surprisal or entropy value to represent higher-order semantic information. Word surprisal is a measure of how unexpected a word is given its preceding linguistic context, and it can be computed using Large Language Models (LLMs), as the negative logarithm of the probability of that word given the previous context. Word entropy, on the other hand, measures how uncertain or unpredictable the next word is. Here, we used a pretrained open-source LLM, Mistral-7B-v0.1 [82] to extract word probabilities, and then computed lexical surprisal and lexical entropy values for each word.
When considering the dynamic nature of the attention switching task, the definition of what context to include for the current word prediction becomes a non-trivial problem. From the machine perspective, given a word w belonging to, e.g., the left stream, the LLM would predict it more easily when provided with all the available linguistic context from the left stream. However, from the neural/behavioral perspective, since participants were flexibly re-orienting their attention between left and right streams, the optimal context would be impacted by the attention switch and, potentially, store information of previously attended blocks. To compare these context-accumulation alternatives, we represented context according to four alternative representations. A machine-ideal model, Oracle, considers as context all words preceding the current word in one particular stream, whether they were attended or unattended. As such, this model is unaware of the switch of attention. Among more neurally-plausible and switch-aware context accumulation models, we constructed the Attention model, which incorporates as context any previously attended block from both left and right speech streams, and the Speaker Specific model, which displays a speech stream bias, whereby only previously attended blocks of the same stream are included as context for lexical prediction of the current block. Similarly to Attention and Speaker Specific, the fourth model, Reset, is switch-aware, but not context-aware. In fact, it does not keep track of any previous speech block, neither attended nor unattended, and instead resets the context following each attention switch cue.
Temporal Response Function (TRF) and analysis procedure