Self-Hosted AI Models: Choose What to Run for Your Work

Running an AI model on your own hardware gives you control over where your data goes, removes per-query costs, and allows offline use. The tradeoff is that you are also responsible for installation, updates, hardware requirements, and the fact that locally-run models are generally smaller and less capable than frontier cloud models. Tools like fenbs keep project context and decisions in a shared record that works with any connected AI assistant, so switching between a local model and a cloud assistant does not mean losing what was established.

This guide covers what self-hosted models are, how to choose one, what hardware you need, and which tasks suit them. It also covers the practical constraints that matter for work use and how to handle context continuity when you move between local and cloud models.

What self-hosted models actually are

A self-hosted AI model is a language model that runs on hardware you control rather than on a vendor's cloud servers. The model weights are downloaded to your machine — or to a server you manage — and inference happens locally. Your queries never leave your hardware.

Most self-hosted models are open-weight models: the weights are publicly released by their developers, which allows anyone to run, modify, and redistribute them. The main families include Meta's Llama series, Mistral AI's models, Google's Gemma, Microsoft's Phi, and Alibaba's Qwen. Each family includes models of various sizes, typically measured in billions of parameters — larger models are more capable but require more memory.

Running these models is easier than it was a few years ago. Tools like Ollama and LM Studio handle the installation, model management, and inference in a form that does not require deep technical expertise. Ollama runs in a terminal and exposes a local API; LM Studio provides a graphical interface for downloading and running models.

Popular self-hosted models compared

Model sizes and hardware requirements change as developers release new quantized versions (compressed models that trade a small amount of capability for lower memory requirements). Check the model's documentation on Hugging Face or the Ollama model library for current hardware recommendations.

Hardware requirements

The minimum to run a useful self-hosted model is a machine with 8 GB of RAM and a modern CPU. That runs smaller 7-8 billion parameter models adequately for simple tasks. Response speed on CPU-only machines is slow — several seconds to tens of seconds per response, depending on the model and task.

A discrete GPU with 8 GB or more of VRAM (NVIDIA GeForce or Quadro, AMD Radeon, or Apple Silicon with unified memory) dramatically improves inference speed. Apple Silicon Macs (M1 and later) with 16 GB or more of unified memory are popular for local inference because the unified memory architecture allows both CPU and GPU to share memory efficiently. For a MacBook Pro with 32 GB of memory, models up to 34 billion parameters run at reasonable speed.

Quantized model variants — compressed versions of a full-precision model — significantly reduce memory requirements. A Q4 or Q4_K_M quantized version of a 7 billion parameter model typically runs in 4-6 GB of RAM, compared to 14 GB or more for the full-precision weights. Most Ollama and LM Studio model libraries offer several quantization levels for each model. Start with a Q4 variant on consumer hardware and test a less compressed version if output quality matters and your hardware can support it.

For demanding tasks or large models, the hardware cost can be significant. The decision to self-host makes more sense if data control is a hard requirement, you have suitable hardware already, or you run enough queries that cloud API costs would otherwise be substantial.

Tasks that suit self-hosted models

Self-hosted models are well-suited for tasks where data sensitivity matters and the task does not require frontier-model capability. Common examples: summarizing internal documents that should not leave your network, classifying or labeling proprietary data, generating draft text for internal review, coding assistance for code that cannot be sent to an external API, and research on confidential topics.

Tasks that benefit less from self-hosting: anything that requires the most capable models available (frontier models like Claude Opus or GPT-4o are significantly more capable than the best currently available open-weight models for complex reasoning), real-time web search, image generation, or multimodal tasks that the chosen local model does not support.

Running models with Ollama

Ollama is a popular tool for running open-weight models locally. After installation, you pull a model with one command and start a chat in the terminal. Ollama also exposes a local API on port 11434 that is compatible with the OpenAI API format, which means many AI tools that support custom API endpoints can connect to a locally-running model through Ollama.

Some AI coding tools — including Cursor — can be configured to use a custom model API endpoint, which allows you to point them at a locally-running model via Ollama. The capability varies by tool and model; check the specific tool's documentation for how to configure a custom endpoint.

LM Studio is an alternative for users who prefer a graphical interface. It provides a model browser for downloading open-weight models, a built-in chat interface, and a local API endpoint compatible with the OpenAI format. Users who find terminal-based tools uncomfortable often try LM Studio first, then add Ollama when they need the command-line API or scripting capabilities. Both tools are free and can be installed on the same machine.

Limitations to be honest about

Self-hosted models are generally less capable than the frontier cloud models at complex reasoning, nuanced writing, and tasks that require broad world knowledge. A 7 billion parameter local model is not a drop-in replacement for Claude Sonnet or GPT-4o. The gap is narrowing as open-weight models improve, but for tasks where output quality is critical, test the local model carefully before relying on it.

You are also responsible for keeping models updated, managing storage (model weights range from a few gigabytes to hundreds of gigabytes), and handling any instability in the inference software. For an individual experimenter, that maintenance is manageable. For an organization, it is a real operational consideration.

How to do this with fenbs

1. Create a free account and set up a project for the work you want to run on local and cloud models. Add the goal, current tasks, and the key decisions that apply to both.
2. Connect Claude or another supported cloud assistant to fenbs over MCP from Integrations. Use the cloud assistant for complex reasoning tasks where capability matters.
3. For local model sessions, open the fenbs project in your browser before starting. Read the current rules, Where we left off, and the task you are working on, then use that as context in your local model session.
4. After a local session, update the fenbs project with what you completed and what the next step is. That record is available for the next session, whether it runs locally or on a cloud model.
5. Record in Decisions and rules which tasks are handled locally (for data sensitivity reasons) and which can use a cloud model. That rule is available to any connected AI assistant at the start of each session.