[FEEDBACK] Better packaging for llama.cpp to support downstream consumers 🤗

[Vaibhavs10]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - brew install llama.cpp works
2. For Linux (CUDA) - they need to clone and install directly from github
3. For Windows - winget (?)

This adds barrier for non technically inclined people specially since in all the above methods users would have to reinstall llama.cpp to get upgrades (and llama.cpp makes releases per commit - not a bad thing, but becomes an issue since you need to upgrade more frequently)

Opening this issue to discuss what could be done to package llama.cpp better and allow users to maybe download an executable and be on their way.

More so, are there people in the community interested in taking this up?

[KhazAkar]

From this, IMO it only misses Linux+CUDA bundle to be useable as download & use.

Click to expand

If we want better packaging on Linux, we can also work on snap/bash installer when trying to use pre-built packages.

[michaelgiba]

On the first point this was a pain point I faced as well (#15249)

[maxwell-kalin]

"snap" the thing from ubuntu?

[mitkox]

It’s high time HuggingFace to copy Ollama’s packaging and GTM strategy, but this time, give credit to llama.cpp. Ideally, we should retain llama.cpp as the core component.

[KhazAkar]

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

[mitkox]

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

[slaren]

Is the barrier the installation process, or the need to use a complex command line to launch llama.cpp?

[ericcurtin]

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

[qnixsynapse]

[slaren]

Are you using the GGML_CPU_ALL_VARIANTS option?

[qnixsynapse]

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

[slaren]

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

[qnixsynapse]

Thank you!

[simonw]

For me the biggest thing is llama-server. That tool is fantastic... but it has very low discoverability. Until a few months ago I still thought it was just a demo because it lived in the examples/ folder (I just noticed it moved from there to the tools/ folder in May).

I'd love to see more emphasis placed on llama-server. It's really good! I think it's probably the most end-user appropriate way to interact with this project.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux, which on Mac and Windows work like a desktop application: you get an icon you can use to launch the tool which start the server running and launch a window that shows that web UI.

Maybe include systray integration and a simple UI for selecting and downloading models too.

At that point llama-server would feel like an alternative to Ollama and LM Studio and Jan. I think it deserves that - it has most of what those tools offer implemented already, what's missing is an installer and a think desktop shell.

[YavorGIvanov]

Agreed. If llama-server was nicely packaged on all OSes and could be easily distributed and run everywhere, then I don't see a reason to use any alternative. The benefit will also be that you get the latest version of llama.cpp immediately without waiting for other companies to package it with nice UI.

[t1u1]

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux,

An easy way to do this is to make it a PWA. Will work in most major browsers, including Safari, Chrome and its derivatives, Android and iOS, but not Firefox. It just takes a few lines of code to do this.

[allozaur]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

[calculatortamer]

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

[red-co]

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

[yaronsumel]

It will be cool if 'llama-server' would have auto configuration option to the machine/model like 'ollama' does it.

[exxocism]

For windows maybe choco and windows store would be a good idea? 🤔

[acbits]

llama.cpp as a project has made LLMs accessible to countless developers and consumers including me. The project has also consistently become faster over time as has the coverage beyond LLMs to VLMs, AudioLMs and more.

One feedback from community we keep getting is how difficult it is to directly use llama.cpp. Often times users end up using Ollama or GUIs like LMStudio or Jan (there's many more that I'm missing). However, it'd be great to offer a path to use llama.cpp in a more friendly and easy way to end consumers too.

Currently if someone was to use llama.cpp directly:

1. For Mac - `brew install llama.cpp` works

2. For Linux (CUDA) - they need to clone and install directly from github

I created a rpm spec to manage installation though I think flatpaks might be more user friendly and distribution agnostic.

[schnow265]

For Windows - winget (?)

The released Windows builds are available via Scoop.

Updates happen automatically. Old installed versions are kept, and current one symlinked into a folder „current“ which provides the executables on the path.

[damianofalcioni]

is it feasible to have a single release for OS including all the backend?

[slaren]

It's technically possible, but the size of the package would be quite big.

[SamuelMarks]

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

[tellsiddh]

For linux I just install the vulkan binaries and run the server from there. Maybe we can have a install script like ollama that detects the system and launches the server which can be controlled from an app as well as cli? The user then gets basic command line utillities like run start stop load list etc?

[leoxzhao]

On Mac, the easiest way (also arguably the safest way) from a user's perspective is to find it in App Store, and install from there. Because of apps from App Store are in a sandbox, so from a user's point of view, installing or uninstalling is simple and clean. Creating a build and passing the App Store review might take some efforts (due to the sandbox constraint), but it should be a one-time thing.

[O-J1]

Its my understanding that none of the automated installs support GPU acceleration. I might be wrong but its definitely the case for Windows, which makes it useless to install via winget.

[slaren]

The winget package comes with the Vulkan backend.

[O-J1]

Does that work reliably with CUDA cards and does it support the same kinds of models (Qwen2.5VL for instance)

[anderspitman]

My experience with the Vulkan backend on a RTX 3060 has been excellent.

[henk717]

To me the biggest advantage ollama currently has is that the optimal settings for a model are bundled, the gguf spec would allow for this to since its versatile enough to make this a metadata field inside the model. It would allow people to load the settings from a gguf and frontends can extract them and adapt them as they see fit. I think that part is going to be more valuable than obtaining the binary since downloading the binary from github is not that hard.

[logan-markewich]

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

[digitalspaceport]

My personal wishlist

1. Llama-swap integration
2. Catchall compiled binaries for linux, win, osx
3. GUI improvements

[neutrinotek]

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

[ericcurtin]

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

[gesrtewrtwr]

I was about to open a similar issue and ask when CUDA Linux builds, like text-generation-webui has them.
(Fortunately the newly introduced --cpu-moe (never tried the same over a regex) increased the tokens per second for my 4070+64GB RAM setup from:
Before: ~6 t/s tg: llama-server -m 'Qwen3-30B-A3B-Instruct-2507-BF16-00001-of-00002.gguf' --n_gpu_layers 7 --port 8080 --threads 3 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 4096 (~85% VRAM usage).
After: ~10 t/s tg: llama-server -m 'Qwen3-30B-A3B-Instruct-2507-BF16-00001-of-00002.gguf' --n_gpu_layers 99 --port 8080 --threads 3 --cpu-moe --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0 -c 4096 (~40% VRAM usage).
Using text-generation-webui' CUDA build I get like 12 t/s, so not bad at all for the -bin-ubuntu-vulkan-x64.zip build so far.)

Another thing is to address the ..ubuntu.. in the release build's name and either rename it to ..linux.., or add linux to the ubuntu name, because the build works on my arch just fine.

[ericcurtin]

Well I guess it's built against Ubuntu libraries so Ubuntu makes sense. This is a Linux packaging problem in general, if using dynamic linking, you gotta build for each distro. But because things are also packaged as containers, you can just use the containers on pretty much any Linux distro.

[gesrtewrtwr]

If it's indeed build against Ubuntu, then I guess it's luck that it runs on Arch, because ldd ./llama-server shows a list, out of which 25 libs are linked against external libs.
AI says Ubuntu has easily on order of magnitude more users. I was hesitant to try the ubuntu-vulkan build because I thought it wouldn't run on Arch (but the "llama.cpp webui is all you need" mentality convinced me after some time to try it first and maybe build later, if it doesn't work). An Arch build would be still nice or ..arch-ubuntu.. in build name.

What's the difference between cudart-llama-bin-win-cuda-12.4-x64.zip and llama-b6210-bin-win-cuda-12.4-x64.zip? Can't find anything in docs.

[ericcurtin]

Yeah, sometimes we can get lucky between distros and binaries are still somewhat ABI compatible cross-distro, but it's probably not something worth depending on.

[anderspitman]

Linux has decent backwards compatibility. Since Arch is rolling release, it tends to have newer versions of libraries than systems where executables are built, which helps things work often.

@gesrtewrtwr couldn't you get much better performance by using a quantized Qwen3-30B-A3B? Sorry if I'm missing something. I'm still pretty new to this.

[LMSSonos]

Using the docker-image llama.cpp:server and mainly running it on my raspberry - therefore missing the latest docker builds for arm64 (issue #13891)

I have tested with different UIs and also ollama and as mentioned from other users, llama-server is kind of hidden but really basic but still awesome, as it covers many of my daily work tasks - recently started to test out the vscode extension, but not managed to bring it up to production

[ericcurtin]

If you install this version of docker:

curl -fsSL https://get.docker.com | sudo bash # installs everything, including docker model runner

https://docs.docker.com/ai/model-runner/get-started/#enable-dmr-in-docker-engine

you'll be back in business with the "docker model runner" server for arm64, (which uses llama-server, but you get model swapping, pushable models to OCI registries, some goodies like this)

[LMSSonos]

If you install this version of docker:

curl -fsSL https://get.docker.com | sudo bash # installs everything, including docker model runner

@ericcurtin thank you for your suggestion, I appreciate it. But for the same reason I don't want to use ollama I also don't want to use docker ;-) I prefer to have less features. But just started to try to compile llama.cpp on my arm64 as suggested by others on the post ;-)

Build worked on raspberrypi, see here #13891 (comment) for the details.

[randomqhacker]

Create a separate installer/launcher per platform that checks CPU/GPU/iGPU against a database and downloads the right executable. The same executable could be used to update. Advanced settings for configuring server options and model parameters. Have a curated list of quantized models to download and launch for that hardware. Have a "custom" option that prompts the user to save a well commented batch file with several examples.

[BradHutchings]

You might want to look into my Mmojo-Server project. One binary built with Cosmopolitan, that runs on ARM/x86 and macoS/Linux/Windows. I have a Mmojo-Server prebuilt on HuggingFace.

I use generic CPU inference, which isn't awful for small models. 12B on good i7/i9. 4B on a Raspberry Pi. While GPU support would be great, it mostly confuses users who are challenged a little by the Terminal.

I even sell a Pi based appliance, if you want something to plug and play. https://Mmojo.net

[06kellyjac]

llama.cpp on NixOS is very easy in my experience.

Simply adding the following to your config gets you llama-server etc.

environment.systemPackages = [
  pkgs.llama-cpp
];

Depending on your config it may already have CUDA enabled. If not you can do

environment.systemPackages = [
  (pkgs.llama-cpp.override { cudaSupport = true; })
];

For rocm

environment.systemPackages = [
  (pkgs.llama-cpp.override { rocmSupport = true; })
];

Vulkan

environment.systemPackages = [
  (pkgs.llama-cpp.override { vulkanSupport = true; })
];

Etc etc.

You can also try combo multiple but your mileage may vary

[SteelPh0enix]

Note that nix flake used for building llama.cpp is currently outdated and lacks some configuration options for ROCm and CUDA builds; i've made a PR to fix that: #15747
(it will have to be synced with nixos repository too)

[06kellyjac]

IDK about the in-repo flake but I'm using the nixpkgs build.

https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/by-name/ll/llama-cpp/package.nix

[jjerphan]

Note that llma.cpp is available on conda-forge, and can be installed with conda, mamba, pixi, etc.

CUDA builds exists for Windows and Linux, and macOS builds are optimized for accelerate and Metal on x86_64 and arm64 respectively.

This distribution can easily be updated by anyone to newer versions and refined for specific targets if needed.

[3Simplex]

I have a good idea for a self configuring front end, just been a little consumed with my new code companion project to start on it.

For Windows users my Llama.Cpp-Toolbox is probably a good place to get up and running fast. I'm going to be updating it soon with more functionality for power users.

The new idea basically would map out the functions after each build, generate a GUI from them and should never go out of date. I haven't thought hard about how it would look but pulling the options and the executables seems easy enough. The scripts and examples could be annoying but if I can determine how to pinpoint them and the instructions or options it could work. I'll get around to it when my new Ai-Code-Companion can scan it all and I'll see what's possible.

eye rolls
Crap that name was already taken, time to rename the project.

[3Simplex]

I solved CURL and dependency management. The script now also installs ccache since that build takes so long without it. The only manual thing you have to do is install either cuda or vulkan but that is said in the GUI...
I have not yet added personal fork source for a toolbox repo/branch but that is on the way, changing llama.cpp repo/branch was always available.

[Rotatingxenomorph]

I had to install extra stuff for compiling with curl to work when compiling CUDA version on ubuntu 25.04 today, it was pretty obscure: sudo apt-get install curl libssl-dev libcurl4-openssl-dev

[ericcurtin]

This is pretty normal and standard for a C/C++ application that uses libcurl as a dependency, you need the headers and library.

[Rotatingxenomorph]

It's a trap for young players, though. You just get told you need curl, so you get curl but that doesn't fix it. Then you google and find a random 2 year old web page to find what you need to do.

[SteelPh0enix]

the thing is that the environment setup process will depend on the specific Linux distribution, keeping a guide for major distros is technically possible, but since most people who'd want to manually compile llama.cpp already know how to set it up it seems unnecessary, considering that you can get the packaged versions in plenty of distributions already...

an alternative is to use docker/nix to build, but that's probably not something a new guy would know how to use either.

[Rotatingxenomorph]

the thing is that the environment setup process will depend on the specific Linux distribution, keeping a guide for major distros is technically possible, but since most people who'd want to manually compile llama.cpp already know how to set it up it seems unnecessary, considering that you can get the packaged versions in plenty of distributions already...

an alternative is to use docker/nix to build, but that's probably not something a new guy would know how to use either.

It is true that by the time you've got to compiling it you can find out how to fix it. I was a bit worried because it was some old web page, but I checked later to see if Claude Opus would fix it and it did. Have to get used to asking AI to solve things as the web kind of dies off.

I still don't know if I even needed to compile it with curl at all, but I just assumed it might be important for the llama-server to work.

[ericcurtin]

@slaren if you build with GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS on. And have all backends installed, is there a way at runtime to switch between say vulkan and ROCm or vulkan and CUDA?

[slaren]

You can use --device to select the devices to use.

[ericcurtin]

Thanks, interestingly my integrated graphics aren't listed as Vulkan-capable:

$ ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)

[slaren]

I believe you need to enable them explicitly by setting GGML_VK_VISIBLE_DEVICES=0,1 etc.

[ericcurtin]

Yup, that worked:

$ GGML_VK_VISIBLE_DEVICES=0,1 ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon Graphics (RADV RENOIR) (10825 MiB, 10825 MiB free)
  Vulkan1: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)

[0cc4m]

This should be fixed by #15947. Can you give it a try?

[Windsage63]

I just went through this, so it's fresh in my mind. I had not used Linux since about 2012, when I changed our in-house servers to Windows server. Running local llm's I have used Text-Generation-WebUI, Ollama, and LM Studio. With the release of GPT-OSS-120b, most of the wrappers did not keep up with the new changes as quickly as I wanted. I run a 5090, and I also wanted to move to the newer torch release. I was doing some small training stuff that needed Triton anyway, so, Linux.

OP lists using winget, in Windows, but the recompiled llama.cpp is Vulcan. The binary distributions stop at 12.4, so if you need the newer copies of torch >12.4, it's source code. Having been away from Linux a long time, setting up WSL and creating everything for a build was a bit of a struggle. I actually had GPT-OSS write me a step by step walk through with copy and paste commands so I wouldn't screw it up.

Once everything was running, it's great, and a significant speedup. I moved my dev work to Linux now. Then I wrote a script (really me and GPT-OSS wrote a script) to fire up llama.cpp, so I could just pick models from an old fashioned script menu. It's crude, but it's never out of date. I posted the script creation script that GPT wrote below because I thought the idea of a self creating script was nice.

make_l-server.md

[am17an]

IMHO the UI and the packaging can be a dedicated separate repo under the ggml umbrella. Llama.cpp should be remain true to its manifesto of "inference at the edge" and continue to be a fast and feature-full backend for edge devices. The impedance mismatch of being both a developer and a consumer product is probably not easy to manage.

[l29ah]

For Linux (CUDA) - they need to clone and install directly from github

llama.cpp is packaged well in Gentoo, no need to do anything fancy.

[MineloaderGuo]

When will llama.cpp have the ability to search on the internet? Without the ability to search for information online and relying solely on existing training data, answering questions is always outdated or biased, and can only be considered a toy with no practical value

[SteelPh0enix]

I recommend setting up OpenWebUI with llama.cpp as back-end for that. You can plug the llama-server via OpenAI API.

[cheery]

I keep my llama.cpp in a podman container. I've setup scripts to build it in a container as well. And I'm using cuda.

Before I picked up llama.cpp I used ollama, found it bit too excessive and non-suited for me. Also saw that most libraries just package up llama.cpp and run my load there. I ended up just using llama.cpp directly through C API with python bindings.

I figure it's nice to build llama.cpp but a bit difficult to pick up because there are so many options and things to look at. Non technical people might have difficulty picking it up just for that. But I personally am here for it.

I find llama.cpp could use improvement on variety of places. One major place is the chat format interfaces. I think the current way chat templates work is too generic because the way this works is that you have an append-only log and you're paying high cost if you splice it. Chat templates on other hand pretend you can add stuff in the middle and they're essentially for-loops over messages list. I'd suggest you would completely change this design.

• have a toplevel prompt.
• Then have message templates for system, user, assistant prompts.
• Likewise have messages for text in middle completion.
• Finally have an assistant-starts-talking-predule...
• ...and a way to parse the assistant output back to corresponding messages.
• and assistant-stops-talking -token sequence.
• Just have a simple templating format that includes&leaves stuff from message if it is present or not.

In practice that covers all message formats because they're forced to be constrained in same way as long as they generate tokens in sequence. Even if the constraint was lifted some day, I think this makes a good format.

[bugparty]

I think we should focus on what it is now, non-tech users will use Ollama or LMStudio anyway.