Why run an LLM locally instead of using a hosted API?

A local LLM keeps data on your own machine, so there is no API provider to trust, no retention policy to read, and no cross-border transfer to justify. It also works offline with reproducible behavior since there are no silent server-side updates, and there is no per-token cost once it is running.

How do you install and run a model with Ollama?

Install Ollama from the official site, then run 'ollama pull llama3.2' to download a small open-weight model and 'ollama run llama3.2' to chat with it fully offline. Ollama uses a GPU automatically if one is present; on CPU-only machines a smaller 1-3 billion parameter model keeps responses quick.

What should never be pasted into any model, local or hosted?

Never paste live credentials and secrets like API keys or passwords, since a local model will not exfiltrate them but they can end up in shell history or saved transcripts. Also avoid real personal data you have no need to process, and never paste model output straight into a shell or query unread.

How do you pick a local model size for your hardware?

1-3B parameter models run comfortably on a laptop with no dedicated GPU and suit summaries, reformatting, and explanations. 7-8B models reason noticeably better but want a GPU or plenty of RAM. Quantised builds (like q4 tags) trade a little quality for much lower memory use, which often decides whether a model fits or swaps to disk.

AI · PRACTICAL

Run an LLM on your own machine.

A model that never sees the network is a model that never leaks your notes. Here is how to stand one up with Ollama, and what to keep off it entirely.

June 28, 20267 min read

There is a specific moment that pushes security people toward local models: you want to paste a log snippet, a config, or an incident timeline into a model to help you think — and you realise that snippet should absolutely not leave your network. A local LLM solves that. It runs on your own hardware, works offline, and nothing you type is sent to anyone. For sensitive drafting, note-taking and documentation, that property alone is worth the setup.

Why local, specifically

Privacy by construction. The data never leaves the machine. There is no API provider to trust, no retention policy to read, no cross-border transfer to justify to a compliance team.
Offline and reproducible. The model works on a plane or in an air-gapped lab, and the same model version gives you the same behaviour tomorrow — no silent server-side updates.
No per-token cost. Once it runs, you can throw as much text at it as your hardware allows without watching a meter.

The trade-off is honest: a model you can run on a laptop is smaller and less capable than the largest hosted models. For summarising notes, reformatting, drafting documentation and rubber-ducking a problem, that gap rarely matters. For frontier reasoning, it does.

Installing Ollama

Ollama is the least-friction way to run open-weight models locally on macOS, Linux and Windows. After installing it from the official site, pulling and running a model is two commands:

# download a small, capable open-weight model
ollama pull llama3.2

# start chatting with it, fully offline
ollama run llama3.2

That is the whole setup. The first command downloads the weights once; after that everything runs locally. If you have a GPU, Ollama uses it automatically; on CPU-only machines a smaller model (1–3 billion parameters) keeps responses quick. You can list what you have pulled with ollama list and remove a model with ollama rm.

Prompts that earn their keep

A local model is a strong writing and thinking aid for security work. A few patterns that consistently help:

Summarise an incident timeline. Paste rough, timestamped notes and ask for a clean chronological summary with an "open questions" section. Good for handoffs.
Explain a config or log line. "Explain what this iptables rule does, line by line, and flag anything that looks overly permissive." You still verify, but it accelerates the read.
Draft documentation. Turn a bulleted list of what you did into a first-draft runbook, then edit. The model handles the boilerplate; you supply the judgement.
Generate study material. Feed it a topic and ask for practice questions — pairs well with our prompts for self-study.

What never goes into any model — local or not

Running locally removes the network-exposure risk, but it does not suspend good judgement. Keep these off the prompt regardless:

Live credentials and secrets. API keys, passwords, private keys. A local model will not exfiltrate them, but they end up in your shell history, scrollback and any saved transcript. Redact first.
Real personal data you have no need to process. Minimising what you handle is a habit worth keeping even when the tool is offline.
Anything you would then paste into a shell or a query unread. Treat model output as a draft by a fast, confident junior: useful, occasionally wrong, never trusted blindly. This is the "insecure output handling" risk from the OWASP LLM list, applied to your own workflow.

Picking a model size for your hardware

The most common early frustration is pulling a model too big for the machine and watching it crawl. A rough guide to match model to hardware:

Model class	Hardware fit
1–3B parameter models	Run comfortably on a modern laptop with no dedicated GPU. Ideal for summaries, reformatting and explanation — the bread-and-butter security-notes tasks.
7–8B models	Want either a GPU or a machine with plenty of RAM and patience. They reason noticeably better and are still very runnable on a decent workstation.
Quantised builds (tags like `q4`)	Trade a little quality for a large drop in memory use, which is often the difference between a model that fits and one that swaps to disk.

Start small, confirm the workflow is useful, then size up only if the quality genuinely falls short. A fast 3B model you actually use beats a sluggish 8B one you avoid. You can keep several pulled at once and switch per task — a small one for quick reformatting, a larger one when you need it to reason.

A note on scope. This is defensive, practical tooling: a model on hardware you control, used to help with your own notes and documentation. It is not a way to process data you are not authorised to handle.

A local LLM will not replace your best hosted model for hard reasoning. But as a private, offline, zero-cost assistant for the daily grind of security writing — summaries, explanations, drafts, study — it is one of the highest-value tools you can set up in ten minutes, and every byte stays on your machine.