Skip to content

Conversation

netrunnereve
Copy link
Collaborator

This basically gets rid of any unsupported devices early on to avoid a crash as my Intel integrated chip with no 16 bit storage got enabled by #15947.

Before:

RADV_PERFTEST=nogttspill ./bin/llama-bench -t 4 -ngl 100 -m llama-2-7b.Q4_0.gguf -r 2
MESA-INTEL: warning: Haswell Vulkan support is incomplete
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = Intel(R) HD Graphics 4600 (HSW GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 0 | bf16: 0 | warp size: 32 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 470 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
ggml_vulkan: device Vulkan0 does not support 16-bit storage.
main: error: failed to load model 'llama-2-7b.Q4_0.gguf'
ERROR:             vkDestroyFence: Invalid device [VUID-vkDestroyFence-device-parameter]
Aborted (core dumped)

After:

RADV_PERFTEST=nogttspill ./bin/llama-bench -t 4 -ngl 100 -m llama-2-7b.Q4_0.gguf -r 2
MESA-INTEL: warning: Haswell Vulkan support is incomplete
ggml_vulkan: Removed 1 unsupported Vulkan devices.
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 470 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 100 |           pp512 |        192.68 ± 0.51 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 100 |           tg128 |         38.23 ± 0.04 |

build: 8a7549fe (6465)

@netrunnereve netrunnereve requested a review from 0cc4m as a code owner September 14, 2025 03:36
@github-actions github-actions bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Sep 14, 2025
Copy link
Collaborator

@0cc4m 0cc4m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix, but I would prefer it during the device selection, instead of removing devices afterwards. I can also do the implementation if you prefer that.

@netrunnereve
Copy link
Collaborator Author

Ok this should do it. I got rid of the warning message as well as I don't really see the point of that here.

Copy link
Collaborator

@0cc4m 0cc4m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good otherwise. I don't have a device to test this.

Copy link
Collaborator

@0cc4m 0cc4m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@0cc4m 0cc4m merged commit cb5bb6c into ggml-org:master Sep 17, 2025
46 of 48 checks passed
angt pushed a commit to angt/llama.cpp that referenced this pull request Sep 17, 2025
* remove unsupported vulkan devices

* make this happen during selection instead

* pass by reference
@netrunnereve netrunnereve deleted the vulkan_unsupported branch September 17, 2025 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants