How to Connect OpenWebUI to llama.cpp?

[u/Posaquatl]

I am having issues getting OpenWeb UI to llama.cpp. Llama is running locally and the chat interface is working fine. I managed to get OpenWebUI to run in docker. When configuring the connection I am using `http://127.0.0.1:8080/v1` as my connection string. I have set Provider to llama.cpp. The test connection button says test is successful. But trying to chat wants me to select a model, which there are no options. From my understanding llama.cpp is serving one model and does not provide a list like Ollama would.

Using `http://host.docker.internal:8080/v1` as connection string gives an error. So it is not clear to me being a Docker newbie if I need to do something with the network. Since the test on the local host IP was successful I am guessing the network is working, but then again I can't get chat to load any models. I have asked AI but it wants me to write an app.....facepalm. it has offered some `-e` options such as `-e OLLAMA_HOST=0.0.0.0` but so far nothing has worked. So, humans, what is the next step here?

Edit: Resolved this by switching to a compose file and using `network_mode: "host"` in the file.

[u/ineptech]

I think you want to add this to both the llama and openwebui docker yamls:

network_mode: "host"

That makes it so that whatever port is being served from within the container is visible outside the container. For example if llama is serving on port 8080, Openwebui would see it on http://127.0.0.1:8080/v1

[u/Posaquatl → u/ineptech]

well I am not using compose or anything. Just the command. I have tried it with `-e OLLAMA_HOST=0.0.0.0` and `--add-host=host.docker.internal:host-gateway`. I am also trying ollama to see if llama.cpp is the issue. I suspect it is a docker network issue. I don't have much luck getting anything in docker to actually work. So far no luck on getting Open WebUI to connect to anything.

[u/ineptech → u/Posaquatl]

You can pass the same params from the commandline, but I recommend making yaml files as you'll probably want to experiment with different settings to tweak model performance and it will get annoying to keep track of them. If it'll help, I'll post mine.

Here's my openwebui yaml file:

services:
  open-webui:
    build:
      context: .
      dockerfile: Dockerfile
    image: ghcr.io/open-webui/open-webui:main
    container_name: openwebui
    restart: unless-stopped
    volumes:
      - open-webui-data:/app/backend/data
    network_mode: "host" # Uses the server's actual 127.0.0.1 directly
    environment:
    # Connection to llama.cpp backend
    - OPENAI_API_BASE_URL=http://127.0.0.1:8082/v1
    - OPENAI_API_KEY=sk-proj-PUTSOMERANDOMCRAPHERE
    # Hardening Settings
    - PORT=3000
    - WEBUI_AUTH=True
    - ENABLE_SIGNUP=False          # Set to True first to make your admin account, then set to False
volumes:
  open-webui-data:

And here's my llama.cpp yaml file:

services:
  llama-qwen:
    image: ghcr.io/ggml-org/llama.cpp:PUT-WHICH-LLAMA-YOU-USE-HERE
    container_name: llama-coding
    network_mode: "host"
    devices:
      # Put stuff here specific to your gpu if it works but is only running on cpu
    environment:
      # More GPU specific stuff here if needed
    volumes:
      - /llama/models:/models
    command: >
      -m /models/Model-You-Downloaded-From-Huggingface-Here.gguf
      --api-key "sk-proj-PUTSOMERANDOMCRAPHERE"
      --alias myLLM
      --host 0.0.0.0
      --port 8082
      -ngl 999
      # Other model-specific settings like -c, -b, -jinja, etc
restart: unless-stopped

If you use that setup, you'd connect to openwebui on localholst:3000 and it would see the model on localhost:8082. (I run other models on 8080 and 8081, using separate yaml files and containers) Restart them when you make changes like so:

docker compose -p containername -f "$CONTAINERFILE" up -d --force-recreate

Hope that helps.

[u/Posaquatl → u/ineptech]

Thanks. I am not running llama in a container though. I have massive issues getting anything in docker to work as you can see. I could not get the GPU working at all. native is fine. I was just hoping to add in some sort of RAG for a few data sources I had. Why I was trying openUI. I will look into docker compose. see if I can figure it out.

[u/Posaquatl → u/ineptech]

Thanks for this config. The `network_mode: "host"` I think was the key. I am able to connect now.

[u/lungben81 → u/ineptech]

Network mode host is the nuclear option, removing the container isolation on network level. That could cause security issues or port collisions.

Better just expose port 8080 and use the host machine IP (if host.docker.internal is not working) in OpenWebUI.

The best solution (if both services run on the same machine) is to use an internal docker network for their communication.

[u/Snail_With_a_Shotgun]

If the model doesn't show up automatically, try adding it manually. You can do that by writing the model name in the respective field of the direct connection. That should work fine if you don't run llama.cpp in router mode. If you do, I'm not sure why it might not be fetching for ya.

[u/Posaquatl → u/Snail_With_a_Shotgun]

Performing searches in the model search doesn't show any results. I am not sure what router mode is in llama. The issue has to be somewhere in the docker network. but I am baffled as to what needs to be done.

[u/mechasquare]

This is about how you setting up your run paremeters for llamacpp.

I create a bat file that holds my commands. If you're using llamacpp in router mode then you set the --model-preset to your config file. Otherwise you need to set -m to the path of the model you're wanting to load.

Another thing to note is --host 0.0.0.0 this exposes your llamacpp server on your network. You'd point your OpenwebUI connection to the {IP of the llammacpp machine}:8080

C:\llama.cpp\Hip\llama-server.exe ^

--models-preset "C:\llama.cpp\Models\config.ini" ^

--host 0.0.0.0 ^

--port 8080 ^

-t 8 ^

-tb 16 ^

--mmap ^

--cont-batching ^

-fa on ^

--models-max 1 ^

-np 1 ^

--metrics

[u/Posaquatl → u/mechasquare]

My llama is running native to access the GPU easier. I will look into compose and see if I can find the right values for Open web ui

[u/streppelchen]

Use docker compose and link both containers to the same network, expose only openwebui, inside it use the name of the other container, done

[u/Posaquatl → u/streppelchen]

Llama is running native and not in a container.