📌 This is the official implementation and benchmark evaluation repository of Unleashing and Benchmarking the Interleaved Cross-modality Comprehension and Generation

[Your introduction content will go here]

. ├── eval/ # Evaluation scripts │ ├── config.py # Configuration settings │ ├── main.py # Main evaluation pipeline │ ├── prompts.py # Evaluation prompts │ ├── summarize.py # Results summarization │ ├── utils.py # Utility functions │ └── vlm_tools.py # GPT implementation ├── infer/ # Inference tools │ ├── case_bagel.py # Example for combined tasks │ ├── case_gpt.py # Example for VQA tasks │ ├── case_step1x.py # Example for image generation │ └── loader.py # Data loading utilities └── vis.ipynb # Visualization notebook

First, prepare the test dataset:

mkdir <YOUR_DATA_PATH> cd <YOUR_DATA_PATH> huggingface-cli download WeiChow/WEAVE --include test/ --repo-type dataset --local-dir . cd test unzip test.zip

‼️: Then edit the ROOT_PATH in loader.py

The test set file format (test.json) is as follows:

⚠️ Note: Image #1 references the first image, starting from 1. This represents the image index, not the conversation turn. It corresponds to the first image in the images array (images[0]). When using multi-turn conversations, each number index should be replaced once with Image #{idx}<image>\n. For single-turn, simply replace directly.

{ "domain": str, "images": [], "chats": [ ] }

The infer/ directory contains tools for inference. To use WeaveBench, you only need to copy loader.py and integrate it with your model.

For your convenience, we provide three example scenarios:

• 🖼️ Image generation inference only (case_step1x.py)
• 💬 VQA task inference only (case_gpt.py)
• 🔄 Combined inference for both image generation and VQA tasks (case_bagel.py)

⚠️ Important: When testing text generation alone, be sure to set the modality to "unified" and skip with if chat[1]['type'] != 'text':. Otherwise, the text's ending index may be incorrect.

Use the eval/ directory for testing scores:

pip3 install -r requirements.txt

Set the GPT config in eval/config.py.

ROOT_PATH = <YOUR_DATA_PATH> OPENAI_MODEL = <MODEL> # like 'gpt-4o-2024-08-06' AZURE_API_KEY = <YOUR KEY> AZURE_ENDPOINT = <YOUR ENDPOINT> # like "https://api.openai.com/v1/chat/completions"

To run evaluation:

python3 eval/main.py --input_dir result/bagel/imgs --output_dir result/bagel/ --mode img python3 eval/summarize.py --metrics_file result/bagel/metrics.jsonl

The input_dir is the location where you saved images after inference (e.g., result/bagel/imgs). The output_dir is where you store GPT evaluation results. The mode parameter has three options: img, txt, or umm. Please select the appropriate evaluation mode.

💡 Tip: Since we require GPT to output in JSON format, some items may not be scored due to format errors. You can rerun the first script to score these items; it will automatically skip already scored content.

WeaveBench evaluates 4 core metrics:

Consider adding your results to our leaderboard at Weave Project Page. Please contact weichow@qq.com for submission details!

@article{ }