How to Add a GUI to Your Local LLM on Linux (Ollama + Open WebUI on Docker)

Welcome back! If you have already read part one of this series on running a local offline LLM, or if you already have one running in your terminal, let's add a beautiful Graphical User Interface (GUI) to it for ease and comfort. If you've had got the chance to use LLM models within the terminal, you already know that switching between multiple models, uploading documents, or even just dealing with readability is a pain.

Tools I used and Why:

1. Open WebUI:
   Unlike Windows and macOS, Ollama doesn't have a native UI for Linux. I had to choose one third-party front-end, and Open WebUI stood out to me. I won't dig deep into all of it's benefits, but one primary thing that caught my eye is it's ability to pull real-time data from the web, along with features like functions and pipelines, which allow you to communicate with APIs, n8n workflows, agents, and much more.
   I also found Msty AI interesting, as it allows you to compare different models side by side. It definitely has many more features I am not aware of yet and would love to try, but for the time being, I'll stick with Open WebUI and proceed with my roadmap. If you try any other similar tools and liked them, I'd love to hear your thoughts in the "comments" section.

2. Docker Engine:
   Open WebUI recommends using Docker, as it's built on a heavy Python FastAPI backend, a Node.js SvelteKit frontend and an embedded ChromaDB vector database. From the just mentioned tech stack, yes I struggled to pronounce "SvelteKit". Whatever it is, if you don't want to mess up your computer's existing code environments, Docker is the best option. Other primary reason is that running it via Docker works better for communicating with automation tools like n8n. It allows Open WebUI container to run in the background when the screen is locked, setting up as startup application during system reboots, and runs silently in the background as a system service.

   However, if you want to run Open WebUI directly on your main OS via an installer file or Python (pip / uv) methods, please refer to "Quick Start" section in Open WebUI Docs.

   Story behind Docker Engine:
   (You can skip this part to go directly to installation)
   Firstly, I would like to clarify that Docker Engine is different from Docker Desktop. In simple words, Docker Desktop bundles Docker Engine with a Graphical User Interface (GUI), while Docker Engine is a command line version that runs directly on the kernel.

   Here's an interesting fact. Initially, Docker was built only for Linux to run directly on kernel, relying heavily on Linux's kernel features (like namespaces, cgroups etc). When Docker Desktop is introduced for Windows and macOS, a light weight Linux distribution (called LinuxKit) Virtual Machine (VM) was used to maintain the same architecture across all platforms. Ultimately, this architecture solves the biggest problem in Software Development which is "It works on my PC" syndrome. While diving deep into this deserves a separate article, I'll conclude this story by saying that Docker Engine uses fewer resources than Docker Desktop because it runs directly on the kernel without a separate VM on top of it.

Back to Adding the GUI Part:

Installation of Docker Engine:

Because Docker maintains it's own repository, installation is not just a simple "sudo apt install DockerEngine". Instead, we use multiple commands to setup a security key, add Docker's repository to package manager's (APT) address book, and finally install the application using apt.

Although Docker's official website has all the commands, I have added comments after the # explaining exactly what we are doing. To setup Docker's apt repository, use the commands below:

1. Adding Docker's official GPG key:
sudo apt update
##Updates existing apt repository to the latest

sudo apt install ca-certificates curl
##Installs curl tool & ca-certificates software package

sudo install -m 0755 -d /etc/apt/keyrings
##Creates a new directory called 'keyrings' with read only permissions in configuration directory for apt package manager

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
##Downloads GPG security key from docker's official URL and saves it to the directory we created

sudo chmod a+r /etc/apt/keyrings/docker.asc
##Permits all users on the system to read the security key

ca-certificates : A package containing security certificates used to check if HTTPS URLs are authentic.
curl : A command line tool to download files from a given URL
-m : A parameter to indicate following numbers/code are used to set security permissions for a file/folder
-d : Indicates directory
chmod = A command that lets you change or update file/folder permissions

2. Adding Docker's repository to Apt address book:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF Types: deb URIs: https://download.docker.com/linux/ubuntu Suites: \((. /etc/os-release && echo "\){UBUNTU_CODENAME:-\(VERSION_CODENAME}") Components: stable Architectures: \)(dpkg --print-architecture) Signed-By: /etc/apt/keyrings/docker.asc EOF

Then run sudo apt update

tee : This command takes input and writes it into a file. It simultaneously displays the written data on the terminal as output, as seen in the picture above.
EOF : 'End Of File' (EOF) indicates to the system to treat everything that comes after it as plain text until EOF appears again.
$() : Anything between these parentheses runs as a command, even when it's under EOF. The command inside detects your Operating System's code name to fetch a docker version that suits your OS. "noble" in the output from the above picture is the code name of my OS.

3. Installing Docker package:
To install the latest version of Docker, run the below command:
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Run the command below to check if Docker is running:
sudo systemctl status docker

If you see it as active, that confirms Docker is installed and running.
If Docker is inactive, run the below command to start it manually:
sudo systemctl start docker
Similarly, you can stop Docker manually with the command:
sudo systemctl stop docker

Pulling Open WebUI Image and Configuring it With Ollama:

1. To pull (download) the Open WebUI image to Docker, run the below command:
   sudo docker pull ghcr.io/open-webui/open-webui:main

   If you use an Nvidia GPU, we pull a different image with the tag "cuda". I will be using the cuda image since I have an Nvidia GPU.
   sudo docker pull ghcr.io/open-webui/open-webui:cuda

   

   You can view the downloaded images in Docker with the command:
   sudo docker images

2. Now we need to run a container from the downloaded Open WebUI image. For now, all you need to know is that an image is a static package that has everything (like filesystems, code, libraries etc) needed to run an application, while a container is a running instance of that image.

   Not only do we run a container, but we also do port binding from the container to our local machine. This allows the container (which is an isolated instance in docker with it's own network and IP) to communicate with our local computer. Our end goal is to make Open WebUI in Docker communicate with the local computer, allowing us to open it in a web browser. Then, we connect Ollama to Open WebUI, ultimately using both through the browser. The image below shows exactly what I mean visually.

   

   To achieve this, run the following command:
   sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

   

   docker ps command shows all running containers. From the above image, you can see my container is running and port binding is also successful.

   Explaining the command:
   docker : Calls the Docker tool.
   run : Runs a new container from the blueprint ghcr.io/open-webui/open-webui:cuda .
   -d : Runs the container in detached (background) mode so your terminal is free to use.
   -p 3000:8080 : Binds your local computer's port 3000 to the container's default port 8080 .
   --add-host : Maps the local machine's gateway IP (host-gateway) to the default name host.docker.internal inside the container.
   -v : Creates a persistent volume in the specified path.
   --name : Names the container as open-webui.
   --restart always : Container automatically restarts after crashes or system reboots.
   cuda : The tag that adds Nvidia GPU support.

   Note: Please replace "cuda" tag with "main" tag if you do not use an Nvidia graphics card.

   Now, you should be able to access Open WebUI from your browser by navigating to http://localhost:3000/ . You need to create an admin account before you see the UI below.

   

3. However, you would not be able to see your downloaded LLM models from Ollama yet. (If you haven't downloaded any model, run ollama pull <modelname> . For a detailed guide, you can refer to part 1 of my series).

   Open WebUI can't detect your local LLM models right now because, by default, Ollama only listens to applications running directly on your local machine. To fix this, you need to edit Ollama's system service file, allowing it to listen to Docker's gateway address.

   By default, Docker's gateway address is usually 172.17.0.1. You can check yours by running the command:
   ip addr show docker0

   Note your Docker gateway address and open Ollama's service file by running:
   sudo systemctl edit ollama.service

   Now update the environment variable to replace the Ollama host address with your Docker's gateway address as shown below:

   

   Note: While replacing the local host IP address with Docker's gateway address is more secure, Ollama will now only listen to Docker, meaning you won't be able to use Ollama directly from your terminal. To make Ollama listen to all available interfaces, use the wildcard address 0.0.0.0. Just keep in mind that this allows anyone on the same local network to access your Ollama instance.

   Now, save the file by pressing Ctrl+O (then Enter), and exit by pressing Ctrl+X.

   Reload the configuration changes and restart Ollama service by running:
   sudo systemctl daemon-reload sudo systemctl restart ollama

4. Refresh http://localhost:3000/ on your web browser, and you should now be able to see all your locally downloaded Ollama models.

   

   You can now run queries on your local LLM Model as seen below:

   

   If you know what "Nemotron" is, by now you might have figured out that the answer from the model is not smart. I was actually asking about Nvidia's Nemotron model, but it hallucinated information about some unknown Nemotron (at least, unknown to me). This makes sense considering that the model doesn't have internet access and it's knowledge cutoff is January 2025. Every offline LLM model has a knowledge cutoff date, making them less than ideal for querying the latest or real-time information.

Can We Add Web Search Capability?

This is the question I asked myself (and then Google) after spending sometime with offline LLMs. My next article will tackle exactly this, and I'm excited to research the options for giving a local LLM web search capabilities. Now, it does make me wonder if the term "Offline AI model" for this series is starting to become absurd... but if it does, so be it!