add benchmark

[huuyu]

Why is the translated model slower than the original model when tested with vLLM?

[huuyu]

export model="outputs/qwen2_5-7B-Instruct-deepseek/"
CUDA_VISIBLE_DEVICES=2 python3 vllm/benchmarks/benchmark_throughput.py
--model $model
--dataset-name sharegpt
--dataset-path test_data/ShareGPT_V3_unfiltered_cleaned_split.json
--trust-remote-code
--dtype bfloat16
--gpu-memory-utilization 0.23
--num-prompts 500

[huuyu]

this is qwen2_5-7B-Instruct:

Throughput: 13.39 requests/s, 5714.22 total tokens/s, 2943.51 output tokens/s
Total num prompt tokens: 103437
Total num output tokens: 109888

this is qwen2_5-7B-Instruct-deepseek/:

Throughput: 9.49 requests/s, 4049.19 total tokens/s, 2085.82 output tokens/s
Total num prompt tokens: 103437
Total num output tokens: 109888

[EricLingRui]

What device are you using? I guess MLA is using a slow kernel.

[Wfd567]

hi ,@huuyu
My test results also became slower after the conversion. Have you solved this problem?