관심 있는 분들을 위해, 3 bit K_XL 양자화 (quantization)에 대한 llama-bench 결과를 공유합니다. 더 개선할 수 있을 것 같지만, 아직까지는 성과가 없었습니다. Command ./llama-bench -m /home/user/llamacpp/modelsmain/unsloth/ds4/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -ngl 21 --split-mode layer -p 512 -n 128 -r 5 Output CUDA Initialization ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72364 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24116 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Benchmark Results Model Size Params Backend NGL Test T/S deepseek4 ?B Q3_K - Medium 119.40 GiB 284.33 B CUDA 21 pp512 116.04 ± 27.64 deepseek4 ?B Q3_K - Medium 119.40 GiB 284.33 B CUDA 21 tg128 7.71 ± 0.09 Build: e3546c794 (9976) System Memory Total Used Free Shared Buff/Cache Available Mem 122Gi 7.9Gi 1.2Gi 165Mi 114Gi Swap 0B 0B 0B NVIDIA-SMI Status +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M.

| |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A | | 58% 55C P2 151W / 390W | 22696MiB / 24576MiB | 35% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3090 Off | 00000000:31:00.0 On | N/A | | 30% 53C P2 120W / 350W | 20193MiB / 24576MiB | 9% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 3090 Off | 00000000:6C:00.0 Off | N/A | | 32% 55C P2 135W / 420W | 17811MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ submitted by /u/consultkitapp [link] [comments]