Yiyang Jiang, Wengyu Zhang, Xulu Zhang, Xiao-Yong Wei, Chang Wen Chen, and Qing Li.
Official Pytorch Implementation of 'Prior Knowledge Integration via LLM Encoding and Pseudo-Event Regulation for Video Moment Retrieval'
Installation | Dataset | Training | Evaluation | Model Zoo
[2024.7.21] Our paper has been accepted by ACM Multimedia 2024 (Oral).
[2024.7.10] The code and dataset of related tasks has been released.
[2024.5.10] The repository is public.
[2024.4.10] The repository is created.
- Clone the repository from GitHub.
git clone https://github.com/fletcherjiang/LLMEPET.git cd LLMEPET- Create conda environment.
conda create -n LLMEPET python=3.8 conda activate LLMEPET- Download the packages
pip install -r requirements.txtFor all datasets, we provide extracted features, download them and place them into features/
. ├── LLMEPET │ ├── llm_epet │ └── data │ └── results │ └── run_on_video │ └── standalone_eval │ └── utils ├── data ├── features │ └── qvhighlight │ └── charades │ └── tacos │ └── tvsum │ └── youtube_uni ├── llama │ └── consolidated.00.pth │ └── tokenizer.model │ └── params.json ├──README.md └── ··· - You can download from huggingface OR
- LLaMA
If you want to try LLaMA-2 or LLaMA-3, you could download the checkpoints from LLaMA-2 or LLaMA-3. You should edit the (llm_epet/llama.py) by yourself.
bash llm_epet/scripts/train.sh bash llm_epet/scripts/charades_sta/train.sh bash llm_epet/scripts/tacos/train.sh bash llm_epet/scripts/tvsum/train_tvsum.sh bash llm_epet/scripts/youtube_uni/train.sh bash llm_epet/scripts/inference.sh results/{direc}/model_best.ckpt 'val' bash llm_epet/scripts/inference.sh results/{direc}/model_best.ckpt 'test' Pack the hl_{val,test}_submission.jsonl files and submit them to CodaLab.
| Dataset | Model file |
|---|---|
| QVHighlights (Slowfast + CLIP) | checkpoints |
| Charades (Slowfast + CLIP) | checkpoints |
| TACoS | checkpoints |
| TVSum | checkpoints |
| Youtube-HL | checkpoints |
If you find the repository or the paper useful, please use the following entry for citation.
@inproceedings{ jiang2024prior, title={Prior Knowledge Integration via {LLM} Encoding and Pseudo Event Regulation for Video Moment Retrieval}, author={Yiyang Jiang and Wengyu Zhang and Xulu Zhang and Xiaoyong Wei and Chang Wen Chen and Qing Li}, booktitle={ACM Multimedia 2024}, year={2024}, url={https://arxiv.org/abs/2407.15051} }