Sound Bridge：Associating Egocentric and Exocentric videos via Audio Cues

Official Pytorch Implementation of 'Sound Bridge：Associating Egocentric and Exocentric videos via Audio Cues'

📢 News

[2025.2.28] The repository is created.

Installation

1. Clone the repository from GitHub.

git clone https://github.com/shhuangcoder/SoundBridge.git
cd SoundBridge

2. Create conda environment.

conda create -n SoundBridge python=3.8
conda activate SoundBridge

3. Download the packages

pip install -r requirements.txt

Dataset

For the dataset, we provide the corresponding download link. Please follow the download instructions provided in the link to download the dataset.
EgoExoLearn: https://github.com/OpenGVLab/EgoExoLearn
CharadesEgo: https://prior.allenai.org/projects/charades-ego
You need to first extract the audio file (.wav) from the video and then use the BEATs model for audio feature extraction. Please acquire the BEATs model according to its requirements. BEATS

The prepared dataset should be in the following structure.

.
├── SoundBridge
│   ├── model
│   └── data
│   └── results
│   └── config
│   └── annotations
│   └── utils
│   └── function
├── datasets
│   └── Charades_Ego
│   │   └── videos
│   │   └── audios
│   │   └── audios_features
│   └── EgoExoLearn
│   │   └── videos
│   │   └── audios
│   │   └── audios_features
├── ckpt
│   └── Cha
│   │   └── association.pt
│   │   └── retrieval.pt
│   └── EgoExo
│   │   └── association.pt
│   │   └── retrieval.pt
│   └── clip_openai_timesformer_base.narrator_rephraser.ep_0001.md5sum_02dbb9.pt
├──README.md
└── ···

Association

Download the Ego4d-pretrained [checkpoint](https://dl.fbaipublicfiles.com/lavila/checkpoints/dual_encoders/ego4d/clip_openai_timesformer_base.narrator_rephraser.ep_0001.md5sum_02dbb9.pth) from [LaViLA](https://github.com/facebookresearch/LaViLa), modify the config file (e.g. ./configs/train_egoonly.yml)
resume: /path/to/your/pretrained_checkpoint/
train: python -m torch.distributed.launch --nproc_per_node=NUM_GPU main_association.py --config ./configs/train_egoexo.yml
test: python main_association.py --config ./configs/test.yml

Retrieval

train: python -m torch.distributed.launch --nproc_per_node=NUM_GPU main_retrieval.py --config ./configs/train_egoexo.yml
test: python main_retrieval.py --config ./configs/test.yml

Model Zoo

• You can download LLaMA from huggingface to generate the required audio description text based on the original video description.
• You can download from huggingface to test the model's performance.

Citation

If you find the repository or the paper useful, please use the following entry for citation.

@inproceedings{huang2025sound,
  title={Sound Bridge: Associating Egocentric and Exocentric Videos via Audio Cues},
  author={Huang, Sihong and Wu, Jiaxin and Wei, Xiaoyong and Cai, Yi and Jiang, Dongmei and Wang, Yaowei},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={28942--28951},
  year={2025}
}

Acknowledgement

The codebase is based on EgoExoLearn. We thank the authors for their efforts.