MapTab is a comprehensive benchmark designed to evaluate the map understanding and spatial reasoning capabilities of Vision-Language Models (VLMs). The benchmark focuses on two core tasks: route planning and map-based question answering, using both metro maps and travel maps.

Project Page: https://ziqiao-shang.github.io/MapTab-Leaderboard/

• Overview
• Task Description
• Dataset
• Version Requirements
• Quick Start
• Supported Models
• Project Structure
• Evaluation Metrics
• Citation

MapTab evaluates VLMs on their ability to:

1. Understand map visualizations - Parse and interpret visual map information
2. Process tabular data - Understand structured information in tables (JSON/CSV)
3. Perform spatial reasoning - Plan routes and answer spatial questions
4. Follow complex constraints - Handle multi-constraint route planning tasks

The benchmark includes various route planning subtasks with different input modalities and constraint levels:

QA tasks evaluate map comprehension across different aspects:

The dataset includes two map types:

• MetroMap: Synthetic metro/subway network maps
• TravelMap: Travel route maps with geographic information

The current release includes the following files under both metromap/ and travelmap/:

• ✅ data/ - Route Planning (RP) task test query set
• ✅ qa_data/ - Question Answering (QA) task query set
• ✅ images/ - Map images
• ✅ prompts/ - Prompt templates for both RP and QA tasks
• ✅ tabulars/ - Edge_tab and Vertex_tab files

Note: In the current release, only the RP task test query set is available. QA task queries and RP task training queries will be released in future updates.

Files in these five folders (data/, qa_data/, images/, prompts/, tabulars/) can be downloaded from Hugging Face: https://huggingface.co/datasets/szq-nju/MapTab

• Python >= 3.8
• PyTorch >= 2.0
• OpenAI SDK (for API-based models)
• vLLM (for local model inference)
• NumPy

export WORKSPACE_DIR="/path/to/MapTab" export API_KEY="your-api-key" # For API-based models

# Generate RP task results bash scripts/generate_rp.sh # Generate QA task results bash scripts/generate_qa.sh

# Evaluate RP task results bash scripts/evaluate_rp.sh # Evaluate QA task results bash scripts/evaluate_qa.sh

The framework has been tested with the following model identifiers in src/generate_lib/utils.py:

• Qwen3-VL-8B-Instruct
• Qwen3-VL-8B-Thinking
• Qwen3-VL-30B-A3B-Thinking
• Qwen2.5-VL-7B-Instruct
• Kimi-VL-A3B-Thinking-2506
• Kimi-VL-A3B-Instruct
• Phi-4-multimodal-instruct
• Phi-3.5-vision-instruct
• Glyph
• Qwen3-VL-2B-Instruct
• llava-v1.6-mistral-7b-hf
• InternVL3_5-30B-A3B
• InternVL3_5-8B
• Ovis2.5-9B

• qwen3-vl-32b-instruct
• qwen3-vl-8b-instruct
• qwen3-vl-30b-a3b-instruct
• qwen3-vl-32b-thinking
• qwen3-vl-8b-thinking
• qwen3-vl-plus
• qwen3-max
• gpt-4.1
• gpt-5
• gpt-4o
• doubao-seed-1-6-251015
• kimi-latest
• GLM-4.1V-9B-Thinking
• GLM-4.6V
• step3
• Qwen3-VL-30B-A3B-Instruct
• gemini-1.5-pro-001
• gemini-1.0-pro-vision-001
• gemini-1.5-flash-001
• gemini-1.5-pro-exp-0801
• gemini-3-flash-preview

Note: For API-based models, we only provide one Aliyun Bailian integration example in src/generate_lib/qwen_api.py. Since API platforms vary, please implement other provider interfaces by following this example.

MapTab/ ├── src/ │ ├── generate.py │ ├── evaluate_planning.py │ ├── evaluate_qa.py │ ├── metromap_utils.py │ ├── travelmap_utils.py │ └── generate_lib/ │ ├── qwen_api.py │ ├── utils.py │ └── vllm_LLMengine.py ├── scripts/ │ ├── generate_rp.sh │ ├── generate_qa.sh │ ├── evaluate_rp.sh │ └── evaluate_qa.sh ├── metromap/ │ ├── data/ │ │ └── test_set/ │ ├── images/ │ ├── prompts/ │ └── tabulars/ ├── travelmap/ │ ├── data/ │ │ └── test_set/ │ ├── images/ │ ├── prompts/ │ └── tabulars/ ├── results/ └── results_evaluate/ 

If you use MapTab in your research, please cite:

@article{shang2026maptab, title={MapTab: Can MLLMs Master Constrained Route Planning?}, author={Shang, Ziqiao and Ge, Lingyue and Chen, Yang and Tian, Shi-Yu and Huang, Zhenyu and Fu, Wenbo and Li, Yu-Feng and Guo, Lan-Zhe}, journal={arXiv preprint arXiv:2602.18600}, year={2026} }