System Prompts for Local LLMs: Roles and Chat Templates

Running a large language model locally -- through Ollama, LM Studio, llama.cpp, or a Hugging Face inference server -- requires the model to receive your system prompt in the exact format it was trained on. Unlike hosted APIs, which handle formatting for you, local setups often require you to apply the correct chat template yourself. Tools like fenbs let you keep the prompts that worked and the lessons learned when they did not, outside any single local environment.

When a local model receives a system message in the wrong format, it usually does not throw an error. It silently ignores the instruction, misparses it as part of the conversation, or responds as if no system prompt was set. This makes testing essential.

How local models handle system messages

Language models are trained on text formatted in a specific way. The training data includes special tokens or markers that separate the system turn, user turns, and assistant turns. At inference time, the input must use the same markers. This formatting specification is called a chat template.

The model does not know what role you intended for a message unless the format tells it. If you send a system instruction without the correct opening and closing tokens for that model, the instruction lands in the wrong part of the input and the model treats it as part of a user message, or ignores it entirely.

Chat templates on Hugging Face

Models published on Hugging Face often include a chat_template field in the tokenizer_config.json file. This Jinja2 template defines how to convert a list of messages into the formatted string the model expects. You can inspect it directly in the model's repository by looking at the tokenizer config.

When you use the Hugging Face transformers library's apply_chat_template() method, it reads this template automatically. You pass your messages as a Python list of dictionaries and get back the correctly formatted string.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("model-name")

messages = [
    {"role": "system", "content": "You are a helpful assistant. Answer in plain English."},
    {"role": "user", "content": "What is a chat template?"}
]

formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
print(formatted)

Print and read the formatted output before running inference. You will see exactly what the model receives, including whether the system message appears, where it appears, and what tokens surround it. This is the fastest way to catch a template problem.

The Hermes prompt format

The Hermes format, used by Nous Research's Hermes series of models and many fine-tunes trained on their data, uses ChatML-style tokens. The system turn is wrapped in <|im_start|>system and <|im_end|>. The same wrapping applies to user and assistant turns.

<|im_start|>system
You are a helpful assistant. Answer in plain English.<|im_end|>
<|im_start|>user
What is a context window?<|im_end|>
<|im_start|>assistant

Many other model families use variations of ChatML, including Qwen models and several Mistral fine-tunes. If you are unsure which format a model expects, look at the tokenizer config on the model's Hugging Face page. Tools like Ollama typically handle this automatically by reading the template from the model file.

Testing that your instruction was applied

The most direct test is to give the model a rule it would not follow by default and then verify compliance. If your system prompt says "always respond with exactly three bullet points," send a varied set of questions and check that every response has three bullet points. Inconsistency means the instruction was not fully applied.

A second test: ask the model to describe its instructions or what it is supposed to do. A model that correctly received your system prompt will describe your rules. A model that did not may describe its generic training or say it has no special instructions.

For deeper detail on how LLMs handle different inputs, what is an LLM covers the fundamentals. The prompt engineering guide applies to both local and hosted models.

Common format errors

• Missing the generation prompt: forgetting add_generation_prompt=True in apply_chat_template() means the model does not know to start generating a response.
• Wrong model family: applying a Llama template to a Qwen model, or vice versa.
• Manual formatting with the wrong tokens: writing the tokens yourself instead of using the template, then getting one character wrong.
• Unsupported system role: some smaller or older models were not trained with a system role. For those, prepend the system content to the first user message.

Saving lessons for next time

Format problems with local models tend to recur, especially when you add a new model or update an existing one. Recording what worked and what failed helps the next session -- or teammate -- skip the debugging step.

How to do this with fenbs

1. Create a free account. Start a personal kanban board for your local LLM work.
2. Add a task for each model you are setting up. In the plan, note the model, the template format, and the inference tool you are using.
3. After testing a prompt with a specific model, record the template and test results in a Lesson in fenbs. Note what worked and what the failure mode was if it did not.
4. Save the working prompt as a Skill in your fenbs project. Include the model name and format so you can retrieve the right version for each model.
5. When you add a new model or update an existing one, check your Lessons first. The template for a model family you have already tested is probably in there.

Related: system prompt examples shows patterns you can adapt for local models. The prompts for LLMs guide covers user-level prompts that work well with local models.