DRAFT: Mistral large 3 Extended Blackwell Support

[hypdeb]

Purpose

Improve performance and support of Mistral Large 3 on Blackwell.

Details

• Added per-tensor scaled Triton configs for MoE (for Eagle draft model)
• (WIP) Added per-block scaled Triton configs for MoE (for target model)
• Added support for Flashinfer TRTLLM per-tensor scaled FP8 MoE kernels (for Eagle draft model)
• (WIP) Added support for Flashinfer TRTLLM per-block scaled FP8 MoE kernels (for target model)
• Fixed Llama4 routing for FP4 MoE
• Added support for Mistral config format in benchmarks/kernels/benchmark_moe.py
• Added support for Mistral tokenizer in vllm/benchmarks/throughput.py

Best Performance Usage

FP8 Checkpoint on DGX B200 (8 devices)

The FP8 model will fit on a single node.
At low concurrencies, deploy with TP8:

VLLM_USE_TRTLLM_RAGGED_DEEPSEEK_PREFILL=1 \ VLLM_ATTENTION_BACKEND=FLASHINFER_MLA \ VLLM_USE_FLASHINFER_MOE_FP8=1 \ VLLM_FLASHINFER_MOE_BACKEND=latency \ vllm serve /models/Mistral-Large-3-675B-Instruct-2512-NVFP4 \ -tp 8 --kv-cache-dtype fp8 --no-enable-prefix-caching \ --config-format mistral --load-format mistral --tokenizer-mode mistral \ --max_model_len 65536 --max_num_seqs 512 --limit-mm-per-prompt '{"image":10}' \ --tool-call-parser mistral --enable-auto-tool-choice 

At higher concurrencies (128 concurrent requests and above), deploy with DP8 and expert parallelism:

VLLM_USE_TRTLLM_RAGGED_DEEPSEEK_PREFILL=1 \ VLLM_ATTENTION_BACKEND=FLASHINFER_MLA \ VLLM_USE_FLASHINFER_MOE_FP8=1 \ VLLM_FLASHINFER_MOE_BACKEND=latency \ vllm serve /models/Mistral-Large-3-675B-Instruct-2512-NVFP4 \ --data-parallel-size 8 --enable-expert-parallel --kv-cache-dtype fp8 --no-enable-prefix-caching \ --config-format mistral --load-format mistral --tokenizer-mode mistral \ --max_model_len 65536 --max_num_seqs 512 --limit-mm-per-prompt '{"image":10}' \ --tool-call-parser mistral --enable-auto-tool-choice 

NVFP4

For NVFP4 checkpoints add the following to leverage the optimized kernels from Flashinfer:

VLLM_NVFP4_GEMM_BACKEND=cutlass VLLM_USE_FLASHINFER_MOE_FP4=1 

With a version of Flashinfer >0.5.3:

VLLM_NVFP4_GEMM_BACKEND=flashinfer-cudnn VLLM_USE_FLASHINFER_MOE_FP4=1 

A bug in the auto-tuner fixed recently (flashinfer-ai/flashinfer#2140) allows using flashinfer-cudnn.

GB200 P/D Disaggregated Dynamo Deployment

There are two options to set up a Dynamo P/D disaggregated deployment of this model. The first one is available immediately and relies on the processing pipeline of Dynamo. The second is pending a PR on Dynamo to enable delegating pre-processing to the vLLM backend.

For compatibility with ToT vLLM, you might need to include some changes that are not currently in upstream Dynamo:

• fix: adjust usage of vLLM for deprecation of 'disable_log_requests' ai-dynamo/dynamo#4659

With Dynamo request processing

Start by copying config.json from Ministral to your model directory.

• Install Dynamo from source to get compatibility with the latest vLLM revision: https://github.com/ai-dynamo/dynamo?tab=readme-ov-file#developing-locally
• Install vLLM on top of that installation

With delegated request processing

Pending on some changes (TODO LINK) in Dynamo, you will be able to skip the file-copying step above.

Next steps

We have identified further optimizations which will be part of other PRs:

• FP8 context attention
• Routing GEMM optimization

Contributors

@dbari, @DanBlanaru, @evezhier, @hypdeb

[mgoin]

It seems there are a few things mixed into this one. Do you think we could prioritize the critical perf features like kernel support and tuned configs?

[mgoin]

I think this isn't a big deal but do we need this dep?

[hypdeb]

We've added some nvtx ranges to the gpu_model_runner.py to make profiling easier, which uses nvtx. This can be split into another PR if it is desirable.

[mgoin]

Can we put this in another PR? We don't want to eat this cost when profiling isn't enabled

[hypdeb]

Will do.

[hypdeb]

It seems there are a few things mixed into this one. Do you think we could prioritize the critical perf features like kernel support and tuned configs?

I'll start by splitting the NVTX stuff out, see how it looks after this.