Choose an Ollama Model Size for a CPU Server

VeloxaHost servers have no GPU, so Ollama loads models into the server's memory and runs them on the CPU. The question is which model fits the RAM you pay for, with room left for its context and for the system. Ollama's library lists the download size of every model tag; this guide turns those sizes into a server size, then checks the choice on the server itself.

It assumes Ollama is installed as in the Ollama self-host guide, with the install script and its systemd service. Prefer one click? Deploy Ollama from the VeloxaHost marketplace.

What you need

• Ollama on Ubuntu 24.04, from the self-host guide. The latest release when checked is 0.40.6. To install exactly that version, set OLLAMA_VERSION, as Ollama's Linux docs describe:

  curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.6 sh
  

• The model's page on ollama.com/library, which lists each tag with its size and context window.

• Free disk for every model you pull. Models are stored in /usr/share/ollama/.ollama/models.

Read the size on the library page

These are the default tags of common models, as Ollama's library lists them on the date above:

Model tag	Download size	Context window
gemma3:270m	292 MB	32K
qwen3:0.6b	523 MB	40K
gemma3:1b	806 MB	32K
llama3.2:1b	1.3 GB	128K
qwen3:1.7b	1.4 GB	40K
llama3.2:3b	2.0 GB	128K
qwen3:4b	2.5 GB	256K
phi4-mini	2.5 GB	128K
gemma3:4b	3.4 GB	128K
mistral:7b	4.4 GB	32K
llama3.1:8b	4.9 GB	128K
qwen3:8b	5.2 GB	40K
gemma3:12b	8.2 GB	128K
qwen3:14b	9.3 GB	40K
gemma3:27b	17 GB	128K

The download size is the least a model needs in memory, not the total. A loaded model also needs memory for its context, and the server needs some for the system and anything else it runs. The context window column is the model's maximum, not what Ollama uses by default.

Pick the quantization by tag

Each model comes in several quantizations, listed on its tags page. Many default tags are 4-bit: llama3.2:3b is the same file as llama3.2:3b-instruct-q4_K_M, 2.0 GB, while the 8-bit llama3.2:3b-instruct-q8_0 is 3.4 GB. For llama3.1:8b the two are 4.9 GB and 8.5 GB. Lower precision means a smaller file and some loss of quality.

Not every default is 4-bit. llama3.2:1b is the 8-bit file at 1.3 GB; the 4-bit llama3.2:1b-instruct-q4_K_M is 808 MB. To choose, pull the full tag:

ollama pull llama3.2:1b-instruct-q4_K_M

Match the model to a server size

Our rule of thumb (ours, not Ollama's): keep the model file at or under about half the server's RAM at the default context, then confirm with the check below.

• Veloxa 1 (1 vCPU, 2 GB RAM, 50 GB disk) at $8 a month, billed at $0.02 an hour and capped at the monthly price: models up to about 1 GB, such as gemma3:1b, qwen3:0.6b or llama3.2:1b-instruct-q4_K_M.
• Veloxa 2 (2 vCPU, 4 GB RAM, 80 GB disk) at $15 a month, billed at $0.03 an hour: 1B to 3B models up to about 2 GB, such as llama3.2:3b, qwen3:1.7b or llama3.2:1b.
• Veloxa 4 (4 vCPU, 8 GB RAM, 160 GB disk) at $30 a month, billed at $0.05 an hour: 4B models such as gemma3:4b, qwen3:4b or phi4-mini, and 7B and 8B models such as mistral:7b or llama3.1:8b with less room to spare.
• Veloxa 8 (8 vCPU, 16 GB RAM, 320 GB disk) at $60 a month, billed at $0.09 an hour: 8B models with a longer context, and 12B to 14B models such as gemma3:12b or qwen3:14b.

Every size is billed by the hour and capped at its monthly price. A 27B model such as gemma3:27b needs more memory than the self-serve sizes offer.

On a CPU, the model's size also sets the speed: a larger model generates fewer tokens per second on the same cores. Measure it, as shown below, before you commit to a model for anything interactive.

Set context and concurrency for your RAM

Four of Ollama's server settings decide how much memory sits on top of the model file:

• OLLAMA_CONTEXT_LENGTH: the default context window. Ollama picks 4096 tokens without a large GPU. A longer context costs more memory.
• OLLAMA_NUM_PARALLEL: requests each model serves at once, 1 by default. Ollama's FAQ says required RAM scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH.
• OLLAMA_MAX_LOADED_MODELS: models loaded at once if they fit, 3 for CPU inference. A request for a model that does not fit waits until an idle one is unloaded.
• OLLAMA_KV_CACHE_TYPE: the context cache's precision. q8_0 uses about half the memory of the default f16 with a very small loss in precision. It applies when Flash Attention is on, which Ollama turns on where the backend supports it; OLLAMA_FLASH_ATTENTION=1 forces it.

Set them in the service with sudo systemctl edit ollama.service, as Ollama's FAQ describes. For example:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify it works

Load the model without a prompt, as Ollama's FAQ shows for preloading, then look at what it takes. free -h is ours:

ollama run llama3.2:3b ""
ollama ps
free -h

In ollama ps, SIZE is the memory the loaded model uses, PROCESSOR should read 100% CPU, and CONTEXT shows the context it was loaded with. In free -h, the "available" column should keep a margin with the model loaded.

Then measure generation speed. The API returns eval_count, the tokens generated, and eval_duration, the time spent generating them in nanoseconds; the calculation is ours:

curl -s http://localhost:11434/api/generate -d '{"model": "llama3.2:3b", "prompt": "Explain DNS in two sentences.", "stream": false}' | python3 -c 'import json, sys; r = json.load(sys.stdin); print(round(r["eval_count"] / r["eval_duration"] * 1e9, 1), "tokens/s")'

Run it a few times with the model loaded. If the number is too low for your use, try a smaller model or a size with more vCPUs.

Troubleshooting

A model will not load, or the service restarts under load. Memory ran out. Pick a smaller tag or quantization, lower OLLAMA_CONTEXT_LENGTH or OLLAMA_NUM_PARALLEL, or move to a larger size. Read journalctl -e -u ollama for the reason.

The first answer after a pause is slow. The model was unloaded: by default it stays in memory for 5 minutes after use. Set OLLAMA_KEEP_ALIVE, for example to 30m, in the same service override.

Answers forget the start of a long conversation. The prompt is longer than the context window, 4096 tokens by default. Raise OLLAMA_CONTEXT_LENGTH if the memory allows.

The disk fills up. Every pulled tag stays on disk. List them with ollama ls and remove unused ones with ollama rm.

FAQ

How much RAM does an 8B model need on a CPU server?

Ollama's library lists llama3.1:8b at 4.9 GB and qwen3:8b at 5.2 GB in their default 4-bit form. Add the context and the system, and a Veloxa 4 (4 vCPU, 8 GB RAM, 160 GB disk) at $30 a month fits one with a modest context. For longer contexts, take a Veloxa 8.

Is the 8-bit version worth the extra memory?

Sometimes. An 8-bit tag is about 1.7 times the size of the 4-bit one and loses less precision. It is also more data to work through for every token, so expect slower answers on a CPU. Try the 4-bit default first and switch only if its answers fall short.

Can I run several models on one server?

Yes, if they fit. On a CPU, Ollama keeps up to 3 models loaded at once when memory allows, and queues requests for a model that does not fit until an idle one is unloaded. Unload one by hand with ollama stop.

Does Ollama use a GPU on VeloxaHost?

No. VeloxaHost servers have no GPUs, so Ollama runs every model on the CPU. That suits small models, batch jobs and internal tools more than fast interactive chat with large models.