vllm-project/vllm Q A · Discussions · GitHub

GitHub discussion is not used anymore, please use the forum for discussion.
Announcements youkaichao

Q&A Discussions

Ask the community for help

You must be logged in to vote

......lib/python3.12/site-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

Rex818 announced May 1, 2025 in Q&A

3
You must be logged in to vote

Running Llama4 quantized on 2xH100 80GB

ilyabcodin announced May 4, 2025 in Q&A · Closed

2
You must be logged in to vote

Determining Overall Speed for One Long Prompt

chigkim announced Apr 30, 2025 in Q&A

8
You must be logged in to vote

Any known integration with n8n ?

wugutech announced Apr 28, 2025 in Q&A

2
You must be logged in to vote

Why temperature=0,top_p=1,seed=42 is still not enough to fix the llm's output!?

beyondguo announced Apr 25, 2025 in Q&A

5
You must be logged in to vote

How to increase context lenght and make things work

kresimirfijacko announced Apr 3, 2024 in Q&A

1
You must be logged in to vote

How does the profile_run work?

zhuangqh announced Nov 7, 2024 in Q&A

9
You must be logged in to vote

How to: custom attention mask? Specifically, bidirectional attention for context.

ojus1 announced Jun 3, 2024 in Q&A

1
You must be logged in to vote

How to only download model without serving it?

yardenshoham announced Dec 26, 2024 in Q&A

2
You must be logged in to vote

Why do vllm set default keep-alive timeout to 5s?

SeungminHeo announced Sep 9, 2024 in Q&A

1
You must be logged in to vote

KV cache in VLLM serve

farouk09 announced Apr 23, 2025 in Q&A

1
You must be logged in to vote

RAG LLM Generating the entire Prompt along with Question in the generated response

Arpx17 announced Feb 29, 2024 in Q&A

4
You must be logged in to vote

Why is the GPU KV cache usage very low?
usage How to use vllm
tammypi announced Jun 18, 2024 in Q&A

4
You must be logged in to vote

Measuring Token Cache Hits in vLLM

LeonardTwcs announced Mar 10, 2025 in Q&A

3
You must be logged in to vote

What is the best way to make DocVLM work with vLLM?

Soum-Soum announced Mar 18, 2025 in Q&A · Closed

2
You must be logged in to vote

Help serving vllm with kuberay

torsteinelv announced Jan 11, 2025 in Q&A

8
You must be logged in to vote

vLLM generating repeated/duplicate responses

saraswatmks announced Jan 21, 2025 in Q&A

8
You must be logged in to vote

Generation speed suddenly became very slow, what might be the cause?

zhuwenfei-wintech announced Sep 27, 2024 in Q&A

3
You must be logged in to vote

LLaVA with GGUF files (and --mmproj argument)?

benmayersohn announced Sep 10, 2024 in Q&A

1
You must be logged in to vote

How to achieve maximum performance?

gaby announced Mar 20, 2025 in Q&A · Closed

1
You must be logged in to vote

working example of vllm using whisper?

silvacarl2 announced Jan 15, 2025 in Q&A · Closed

4
You must be logged in to vote

MultiModal model batched inference

aswad546 announced Feb 24, 2025 in Q&A

3
You must be logged in to vote

Is there a way to set max-concurrency

tommasofavaron1 announced Jan 31, 2024 in Q&A

2
You must be logged in to vote

How can I log user id from the API request

vinayK34 announced Feb 6, 2025 in Q&A

1
You must be logged in to vote

Can vllm serving clients by using multiple model instances?

aoyulong announced Jun 21, 2023 in Q&A

7