How to Deploy an LLM Locally with Ollama: A Step-by-Step Guide

Ollama has emerged as the leading tool for deploying Large Language Models on personal hardware. It abstracts away the complexities of model quantization and GPU management, providing a straightforward CLI and a local REST API. This guide walks you through the deployment process from zero to an active inference endpoint.

Ollama is compatible with macOS, Linux, and Windows. Ensure your machine has adequate RAM—8GB is the minimum for most 7B parameter models, while 32GB or more is recommended for 34B models and above. The entire setup takes only a few minutes.

Article illustration

Step 1: Install Ollama

Download the installer directly from the official Ollama website for Windows or macOS. On Linux, use the standard installation script:

curl -fsSL https://ollama.com/install.sh | sh

After installation, verify it works by checking the version: ollama --version.

Step 2: Pull and Run a Model

Ollama hosts a wide range of models on its library. To deploy Meta’s Llama 3.1, simply run:

ollama run llama3.1:8b

This single command downloads the model weights and launches an interactive terminal chat. Models are automatically optimized for your CPU/GPU using llama.cpp.

Step 3: Serve via the REST API

For application integration, Ollama runs a server at http://localhost:11434. You can send prompts using curl:

curl -X POST http://localhost:11434/api/generate -d '{"model": "llama3.1:8b", "prompt": "Hello, world!"}'

This API endpoint allows you to build front-end interfaces or integrate the LLM into existing software pipelines without custom networking code.

Conclusion

Deploying an LLM with Ollama transforms complex AI infrastructure into a single command. It is the most efficient way to gain hands-on experience with open-source models while retaining complete data privacy. Experiment with different models from the library to see which one best fits your specific use case.

sarah antaboga
Author: sarah antaboga

Leave a Reply

Your email address will not be published. Required fields are marked *