Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation short-term and long-term audio information, as well audio-visual interaction. Unlike the prior work where systems make decision instantaneously using features, we propose novel framework, named TalkNet, that makes by taking both features into consideration. TalkNet consists temporal encoders for feature representation, cross-attention...

Emotion represents an essential aspect of human speech that is manifested in prosody.Speech, visual, and textual cues are complementary communication.In this paper, we study a hybrid fusion method, referred to as multi-modal attention network (MMAN) make use visual emotion recognition.We propose novel multimodal mechanism, cLSTM-MMA, which facilitates the across three modalities selectively fuse information.cLSTM-MMA fused with other uni-modal subnetworks late fusion.The experiments show...

A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker mixture. The prior studies focus mostly on highly overlapped However, target-interference overlapping ratios could vary over wide range 0% 100% in natural communication, furthermore, be absent mixture, mixtures such universal scenarios are described as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">general mixtures</i> . requires an auxiliary...

Prior studies on audio-visual speech recognition typically assume the visibility of speaking lips, ignoring fact that visual occlusion occurs in real-world videos, thus adversely affecting performance. To address this issue, we propose a framework restores occluded lips video by utilizing both itself and corresponding noisy audio. Specifically, aims to achieve these three tasks: detecting frames, masking areas, reconstruction masked regions. We tackle first two issues Class Activation Map...

Most of the prior studies in spatial Direction Arrival (DoA) domain focus on a single modality. However, humans use auditory and visual senses to detect presence sound sources. With this motivation, we propose neural networks with audio signals for multi-speaker localization. The heterogeneous sensors can provide complementary information overcome uni-modal challenges, such as noise, reverberation, illumination variations, occlusions. We attempt address these issues by introducing an...

A speaker extraction algorithm seeks to extract the speech of a target from multi-talker mixture when given cue that represents speaker, such as pre-enrolled utterance, or an accompanying video track. Visual cues are particularly useful is not available. In this work, we don't rely on speaker's speech, but rather use face track cue, referred auxiliary reference, form attractor towards speaker. We advocate temporal synchronization between and its lip movements direct dominant audio-visual...

Recently, the application of diffusion probabilistic models has advanced speech enhancement through generative approaches. However, existing diffusion-based methods have focused on generation process in high-dimensional waveform or spectral domains, leading to increased complexity and slower inference speeds. Additionally, these primarily modelled clean distributions, with limited exploration noise thereby constraining discriminative capability for enhancement. To address issues, we propose...

Decoding speech from brain signals is a challenging research problem that holds significant importance for studying processing in the brain. Although breakthroughs have been made reconstructing mel spectrograms of audio stimuli perceived by subjects at word or letter level using noninvasive electroencephalography (EEG), there still critical gap precisely continuous features, especially minute level. To address this issue, paper proposes State Space Model (SSM) to reconstruct spectrogram EEG,...

Speaker extraction algorithm relies on the speech sample from target speaker as reference point to focus its attention. Such a is typically pre-recorded. On other hand, temporal synchronization between and lip movement also serves an informative cue. Motivated by this idea, we study novel technique use speech-lip visual cues extract directly mixture during inference time, without need of pre-recorded speech. We propose multi-modal network, named MuSE, that conditioned only image sequence....

Speaker extraction seeks to extract the clean speech of a target speaker from multi-talker mixture speech. There have been studies use pre-recorded sample or face image as cue. In human communication, co-speech gestures that are naturally timed with also contribute perception. this work, we explore sequence, e.g. hand and body movements, cue for extraction, which could be easily obtained low-resolution video recordings, thus more available than recordings. We propose two networks using...

End-to-end time-domain speech separation with masking strategy has shown its performance advantage, where a 1-D convolutional layer is used as the encoder to encode sliding window of waveform latent feature representation, i.e. an embedding vector. A large leads low resolution in processing, on other hand, small offers high but at expense computational cost. In this work, we propose graph encoding technique model fine structural knowledge samples reasonable size. Specifically, build...

Target speaker extraction aims to extract the speech of a specific from multi-talker mixture as specified by an auxiliary reference. Most studies focus on scenario where target is highly overlapped with interfering speech. However, this only accounts for small percentage real-world conversations. In paper, we aim at sparsely scenarios in which reference needs perform two tasks simultaneously: detect activity and disentangle active any We propose audio-visual model named ActiveExtract,...

Neuro-steered speaker extraction aims to extract the listener's brainattended speech signal from a multi-talker signal, in which attention is derived cortical activity. This activity usually recorded using electroencephalography (EEG) devices. Though promising, current methods often have high confusion error, where interfering extracted instead of attended speaker, degrading listening experience. In this work, we aim reduce error neuro-steered model through jointly fine-tuned auxiliary...

Speaker tracking plays a significant role in numerous real-world human robot interaction (HRI) applications. In recent years, there has been growing interest utilizing multi-sensory information, such as complementary audio and visual signals, to address the challenges of speaker tracking. Despite promising results, existing approaches still encounter difficulties accurately determining speaker's true location, particularly adverse conditions speech pauses, reverberation, or occlusions,...

The prevailing noise-resistant and reverberation-resistant localization algorithms primarily emphasize separating providing directional output for each speaker in multi-speaker scenarios, without association with the identity of speakers. In this paper, we present a target algorithm selective hearing mechanism. Given reference speech speaker, first produce speaker-dependent spectrogram mask to eliminate interfering speakers' speech. Subsequently, Long-Short-Term Memory (LSTM) network is...

The speaker extraction algorithm extracts the target speech from a mixture containing interference and background noise. process sometimes over-suppresses extracted speech, which not only creates artifacts during listening but also harms performance of downstream automatic recognition algorithms. We propose hybrid continuity loss function for time-domain algorithms to settle over-suppression problem. On top waveform-level used superior signal quality, i.e., SI-SDR, we introduce...

Speaker extraction seeks to extract the target speech in a multitalker scenario given an auxiliary reference.Such reference can be auditory, i.e., pre-recorded speech, visual, lip movements, or contextual, phonetic sequence.References different modalities provide distinct and complementary information that could fused form top-down attention on speaker.Previous studies have introduced visual contextual single model.In this paper, we propose two-stage time-domain visual-contextual speaker...