A5004717608- Multimodal Machine Learning Applications
- Speech Recognition and Synthesis
- Music and Audio Processing
- Speech and Audio Processing
- Natural Language Processing Techniques
- Video Analysis and Summarization
- Speech and dialogue systems
- Advanced Image and Video Retrieval Techniques
- Domain Adaptation and Few-Shot Learning
- Human Pose and Action Recognition
- Topic Modeling
- Subtitles and Audiovisual Media
- Phonetics and Phonology Research
- Music Technology and Sound Studies
- Digital Communication and Language
- Image and Signal Denoising Methods
- Hearing Loss and Rehabilitation
- Language Development and Disorders
- Time Series Analysis and Forecasting
- Generative Adversarial Networks and Image Synthesis
- Stuttering Research and Treatment
- Robotics and Sensor-Based Localization
- Advanced Data Compression Techniques
- Functional Brain Connectivity Studies
- Video Surveillance and Tracking Methods
The University of Texas at Austin
2021-2025
Massachusetts Institute of Technology
2012-2021
Moscow Institute of Thermal Technology
2020
University of Illinois Urbana-Champaign
2010
Urbana University
2010
Multi-modal learning from video data has seen increased attention recently as it allows training of semantically meaningful embeddings without human annotation, enabling tasks like zero-shot retrieval and action localization. In this work, we present a multi-modal, modality agnostic fusion transformer that learns to exchange information between multiple modalities, such video, audio, text, integrate them into fused representation in joined multi-modal embedding space. We propose train the...
We summarize the accomplishments of a multi-disciplinary workshop exploring computational and scientific issues surrounding zero resource (unsupervised) speech technologies related models early language acquisition. Centered around tasks phonetic lexical discovery, we consider unified evaluation metrics, present two new approaches for improving speaker independence in absence supervision, evaluate application Bayesian word segmentation algorithms to automatic subword unit tokenizations....
In this paper, we present a model which takes as input corpus of images with relevant spoken captions and finds correspondence between the two modalities. We employ pair convolutional neural networks to visual objects speech signals at word level, tie together an embedding alignment learns joint semantic space over both evaluate our using image search annotation tasks on Flickr8k dataset, augmented by collecting 40,000 Amazon Mechanical Turk.
Given a collection of images and spoken audio captions, we present method for discovering word-like acoustic units in the continuous speech signal grounding them to semantically relevant image regions. For example, our model is able detect instances word ‘lighthouse’ within an utterance associate with regions containing lighthouses. We do not use any form conventional automatic recognition, nor text transcriptions or linguistic annotations. Our effectively implements language acquisition,...
Current methods for learning visually grounded language from videos often rely on text annotation, such as human generated captions or machine automatic speech recognition (ASR) transcripts.In this work, we introduce the Audio-Video Language Network (AVLnet), a self-supervised network that learns shared audiovisual embedding space directly raw video inputs.To circumvent need learn audio-visual representations randomly segmented clips and their audio waveforms.We train AVLnet HowTo100M, large...
In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our is capable capturing both word-level and sub-word units, depending on how it configured. What differentiates paper from prior work speech unit the choice training objective. Rather than using reconstruction-based loss, use discriminative, multimodal grounding objective which forces learned to be useful semantic...
In this paper, we propose a simple yet powerful improvement over the recent Self-Supervised Audio Spectrogram Transformer (SSAST) model for speech and audio classification.Specifically, leverage insight that SSAST uses very high masking ratio (75%) during pretraining, meaning vast majority of self-attention compute is performed on mask tokens.We address by integrating encoder-decoder architecture from Masked Autoencoders are Scalable Vision Learners (MAE) into SSAST, where deep encoder...
Multimodal self-supervised learning is getting more and attention as it allows not only to train large networks without human supervision but also search retrieve data across various modalities. In this context, paper proposes a framework that, starting from pre-trained backbone, learns common multimodal embedding space in addition sharing representations different modalities, enforces grouping of semantically similar instances. To end, we extend the concept instance-level contrastive with...
When people observe events, they are able to abstract key information and build concise summaries of what is happening. These include contextual semantic describing the important high-level details (what, where, who how) observed event exclude background that deemed unimportant observer. With this in mind, descriptions generate for videos different dynamic events can greatly improve our understanding interest each video. be captured captions provide expanded attributes video labeling (e.g....
Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, generalization abilities learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation unimodal audio/visual bimodal fusion 7 datasets covering 5 audio-visual tasks in speech audio processing. We evaluate recent...
In this paper, we explore the learning of neural network embeddings for natural images and speech waveforms describing content those images. These are learned directly from without use linguistic transcriptions or conventional recognition technology. While prior work has investigated setting in monolingual case using English data, represents first effort to apply these techniques languages beyond English. Using spoken captions collected Hindi, show that same model architecture can be...
Wei-Ning Hsu, David Harwath, Tyler Miller, Christopher Song, James Glass. Proceedings of the 59th Annual Meeting Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
OursFigure 1: HuBERT: sum of attention weights each frame receives from other frames.Ours (VG-HuBERT3): the [CLS A] token.Attention different heads are coded with colors.
Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed data is costly. Therefore, we propose Speech-CLIP, novel framework bridging and through images to enhance without transcriptions. We leverage state-of-the-art pre-trained HuBERT CLIP, aligning them via paired spoken captions minimal fine-tuning. SpeechCLIP outperforms prior on image-speech retrieval performs zero-shot speech-text direct supervision from Moreover, can...
Reverberation not only degrades the quality of speech for human perception, but also severely impacts accuracy automatic recognition. Prior work attempts to remove reverberation based on audio modality only. Our idea is learn dereverberate from audio-visual observations. The visual environment surrounding a speaker reveals important cues about room geometry, materials, and location, all which influence precise effects. We introduce Visually-Informed Dereverberation Audio (VIDA), an...
In this paper, we investigate the manner in which interpretable sub-word speech units emerge within a convolutional neural network model trained to associate raw waveforms with semantically related natural image scenes. We show how diphone boundaries can be superficially extracted from activation patterns of intermediate layers model, suggesting that may leveraging these events for purpose word recognition. present series experiments investigating information encoded by events.
We present VoiceCraft-Dub, a novel approach for automated video dubbing that synthesizes high-quality speech from text and facial cues. This task has broad applications in filmmaking, multimedia creation, assisting voice-impaired individuals. Building on the success of Neural Codec Language Models (NCLMs) synthesis, our method extends their capabilities by incorporating features, ensuring synthesized is time-synchronized expressively aligned with movements while preserving natural prosody....