How to Host AI Locally for Writing, Research, and Coding

Hosting AI locally means downloading model weights to your own machine and running inference there, so your prompts and responses never leave your hardware. The three components you need are a model, an inference server that runs it, and a way for your tools to talk to the server.

A local setup gives you privacy and no per-query costs, but it requires hardware capable of running the model you choose, and the models available locally are not always as capable as the latest hosted versions. Tools like fenbs can act as an optional cloud record of approved project context that local assistants connect to alongside their local setup, if the client supports MCP.

The components of a local AI setup

Every local AI setup has the same three layers. The model is the file of weights you download — a specific version of Llama, Mistral, Qwen, or another open model. The inference server is the program that loads those weights and runs them to produce responses. The client is whatever you use to send prompts: a browser interface, a coding tool, or a script.

Most people start with Ollama because it combines model management and inference into one command-line tool with a simple API. LM Studio adds a desktop GUI and is useful if you prefer a visual interface for browsing and loading models.

What hardware you need

Smaller models (around 7 billion parameters) run acceptably on most modern laptops with at least 8 GB of RAM if you use CPU inference, though responses are slower. A dedicated GPU with sufficient video memory produces much faster responses — the model fits entirely in VRAM and inference speed improves significantly.

Larger models need more VRAM or system RAM. A model's page on its download source (such as Hugging Face or the Ollama model library) typically notes the recommended hardware. Start with the smallest version that suits your task and test before committing to a larger download.

If your hardware is limited, a quantized version of a model uses less memory at a small cost to output quality. Most inference servers support common quantization formats without extra configuration.

Install an inference server and pull a model

1. Install Ollama from its official site (ollama.com). It runs on macOS, Linux, and Windows and starts automatically as a background service.
2. Open a terminal and run ollama pull llama3 (or another model name from the Ollama library) to download a model. This may take several minutes depending on the model size and your connection.
3. Test the model directly in the terminal with ollama run llama3 and type a simple prompt. If you get a coherent response, the server is working.
4. The server exposes an API at http://localhost:11434 by default. Any client that supports the Ollama API or the OpenAI-compatible endpoint can connect to it.
5. Stop the interactive session with /bye and let Ollama continue running as a background service for other clients.

Add a web interface for easier use

The command-line interface works for testing, but most people prefer a browser-based chat for daily use. Open WebUI is a self-hosted web interface that connects to Ollama, shows your available models, and lets you switch between them in a chat-style view. Install it following its documentation on GitHub and access it at http://localhost:3000 by default.

AnythingLLM is another option that adds document upload and simple RAG (retrieval-augmented generation, where the assistant searches your own documents before answering). Both projects are open source. Check each project's own documentation for current installation steps.

Connect your writing or coding tools

Coding tools that support a local model endpoint — including Cursor, some VS Code extensions, and Obsidian AI plugins — can point to your Ollama server instead of a cloud API. In most cases, set the model endpoint to http://localhost:11434 and pick the model name from your list.

Keep in mind that local models vary in capability. For complex coding tasks, a smaller local model may not produce the same quality as a larger hosted model. Test on representative prompts before switching your whole workflow.

For using AI effectively in everyday work regardless of which model you run, see how to use AI at work.

How to do this with fenbs

1. Create a free account and add the project you are working on to a kanban board.
2. Connect a supported client (Cursor, Codex CLI, or another MCP-compatible tool) to fenbs over MCP from /integrations. Point that client at your local Ollama model for inference.
3. Add the project's current goals, standing rules, and agreed decisions to fenbs. The assistant reads them before it starts, even though it runs on your local model.
4. Save progress in fenbs before you close the session. The local model will not remember anything, but the record in fenbs will.
5. If you switch to a hosted model later, the same fenbs record is available to any connected assistant without copying anything across.

For an overview of the AI tools landscape, see what an AI brain is. For understanding Cursor, see what is Cursor AI.