뚜렷한 설명 없이 한동안 이 문제를 겪어왔습니다. 저의 기본 TPS (Tokens Per Second)는 텐서 병렬화 (Tensor Parallelism) 환경에서 약 39.7tps이며, --sm layer 사용 시 약 31에서 33tps 정도입니다. 하지만 MTP (Multi-Token Prediction)를 사용하면 TPS가 27 tps에서 최대 34 tps 사이를 심하게 오갑니다.

Dual 3090 MTP 미사용:
0.33.712.373 I slot print_timing: id 3 | task 0 | n_decoded = 120, tg = 39.93 t/s, tg_3s = 39.93 t/s
0.36.716.476 I slot print_timing: id 3 | task 0 | n_decoded = 239, tg = 39.77 t/s, tg_3s = 39.61 t/s
0.39.719.036 I slot print_timing: id 3 | task 0 | n_decoded = 359, tg = 39.84 t/s, tg_3s = 39.97 t/s
0.42.720.932 I slot print_timing: id 3 | task 0 | n_decoded = 478, tg = 39.79 t/s, tg_3s = 39.64 t/s
0.45.744.765 I slot print_timing: id 3 | task 0 | n_decoded = 598, tg = 39.77 t/s, tg_3s = 39.68 t/s
0.48.745.675 I slot print_timing: id 3 | task 0 | n_decoded = 717, tg = 39.75 t/s, tg_3s = 39.65 t/s

Dual 3090 MTP 사용:
0.29.114.038 I slot print_timing: id 3 | task 0 | n_decoded = 101, tg = 33.09 t/s, tg_3s = 33.09 t/s
0.32.125.448 I slot print_timing: id 3 | task 0 | n_decoded = 204, tg = 33.64 t/s, tg_3s = 34.20 t/s
0.35.142.644 I slot print_timing: id 3 | task 0 | n_decoded = 302, tg = 33.26 t/s, tg_3s = 32.48 t/s
0.38.144.384 I slot print_timing: id 3 | task 0 | n_decoded = 386, tg = 31.95 t/s, tg_3s = 27.98 t/s
0.41.181.756 I slot print_timing: id 3 | task 0 | n_decoded = 470, tg = 31.08 t/s, tg_3s = 27.66 t/s
0.43.591.768 I slot print_timing: id 3 | task 0 | draft acceptance = 0.56186 ( 218 accepted / 388 generated), mean len = 2.22

이는 단일 3090에서 MTP 사용 여부에 따라 성능이 33tps의 기준점에서 41에서 37tps 사이로 뛰어오르는 것(시간이 지남에 따라 저하되는 것으로 보임)과는 정반대의 동작입니다.

단일 3090 MTP 미사용:
13.16.249.338 I slot print_timing: id 0 | task 2 | n_decoded = 100, tg = 33.28 t/s, tg_3s = 33.28 t/s
13.19.263.720 I slot print_timing: id 0 | task 2 | n_decoded = 200, tg = 33.23 t/s, tg_3s = 33.17 t/s
13.22.277.954 I slot print_timing: id 0 | task 2 | n_decoded = 300, tg = 33.21 t/s,

tg_3s = 33.18 t/s 13.25.281.152 I slot print_timing: id 0 | task 2 | n_decoded = 399, tg = 33.15 t/s, tg_3s = 32.96 t/s 13.28.303.309 I slot print_timing: id 0 | task 2 | n_decoded = 499, tg = 33.14 t/s, tg_3s = 33.09 t/s 13.31.311.098 I slot print_timing: id 0 | task 2 | n_decoded = 598, tg = 33.10 t/s, tg_3s = 32.91 t/s 13.34.312.824 I slot print_timing: id 0 | task 2 | n_decoded = 697, tg = 33.08 t/s, tg_3s = 32.98 t/s 13.37.319.058 I slot print_timing: id 0 | task 2 | n_decoded = 796, tg = 33.06 t/s, tg_3s = 32.93 t/s 13.40.343.944 I slot print_timing: id 0 | task 2 | n_decoded = 895, tg = 33.03 t/s, tg_3s = 32.73 t/s Single 3090 with MTP: 0.28.098.655 I slot print_timing: id 0 | task 0 | n_decoded = 138, tg = 45.49 t/s, tg_3s = 45.49 t/s 0.31.142.845 I slot print_timing: id 0 | task 0 | n_decoded = 242, tg = 39.82 t/s, tg_3s = 34.16 t/s 0.34.161.896 I slot print_timing: id 0 | task 0 | n_decoded = 353, tg = 38.81 t/s, tg_3s = 36.77 t/s 0.37.215.826 I slot print_timing: id 0 | task 0 | n_decoded = 453, tg = 37.28 t/s, tg_3s = 32.74 t/s 0.40.236.317 I slot print_timing: id 0 | task 0 | n_decoded = 556, tg = 36.65 t/s, tg_3s = 34.10 t/s 0.43.270.871 I slot print_timing: id 0 | task 0 | n_decoded = 668, tg = 36.69 t/s, tg_3s = 36.91 t/s 0.46.290.045 I slot print_timing: id 0 | task 0 | n_decoded = 772, tg = 36.37 t/s, tg_3s = 34.45 t/s 0.46.341.870 I slot print_timing: id 0 | task 0 | draft acceptance = 0.56280 ( 354 accepted / 629 generated), mean len = 2.29 My command: C:\llama-cpp\llama-cuda\llama-server.exe -m C:\llama-cpp extbackslash models extbackslash gemma-4-31B-it-QAT-Q4_0.gguf --mmproj C:\llama-cpp extbackslash models extbackslash mmproj-gemma-4-31B-it-QAT-BF16.gguf --model-draft C:\llama-cpp extbackslash models extbackslash mtp-gemma-4-31B-it.gguf --spec-type draft-mtp --spec-draft -ngl 99 --spec-draft-n-max 6 --spec-draft-p-min 0.75 --cache-type-k q8_0 --cache-type-v q8_0 -fa on --ctx-size 200000 --temp 0.3 --host 0.0.0.0 --port 8080 --top-p 0.95 --top-k 64 --split-mode tensor --tensor-split 1,1 --kv-unified submitted by

/u/DjCanalex [link] [comments]