System Prompt Best Practices: Guidance, Not Enforcement

System prompts usually steer the model well, but nothing makes their rules hold every time. A pushy user, or app code that does whatever the model asks, can break a rule the prompt states clearly. Good prompt writing helps a lot, but it cannot replace checks in code.

Picture a common setup. A savings app in Chennai adds a chat helper that answers questions about each user's savings goals, like Rs 500 a week for a Diwali trip. The setup is made up, but the problem is real.

The compliance lead, the person who makes sure the company follows the law, sets two rules that must never break. The helper must never give personalised investment advice. It must never show one user's data to another user.

The engineer writes both rules into the system prompt. In the team's own tests, the helper obeys every time. Before launch, the compliance lead spends an afternoon trying to break it with test accounts.

Here is what happens, in made-up numbers. Out of 60 tricky chats, the helper names a specific mutual fund in 3. A mutual fund is a pool of many people's money that a company invests for them.

In another chat, the compliance lead asks for the balance on account ending 7721. The helper asks its lookup tool for account 7721, and the tool returns it. Another test user's balance appears on screen: Rs 18,400.

The prompt was well written. The problem was where the rules lived: the prompt asked the model to behave, and no code made sure it did. This guide shows how to write a good prompt, why it still fails, and which rules belong in code.

Six terms carry the rest of this guide. Each gets a plain meaning and an example from the savings helper.

• Model: the program that reads text and writes text back. It reads a question about a Diwali goal and writes an answer.
• System prompt: the instructions your app sends to the model before every conversation. Users never type it and usually never see it. Some research papers call it the system message.
• User message: what the person types in the chat box. Your app sends it to the model together with the system prompt.
• Tool: an action your app lets the model ask for, like a button it may press. The model only asks; your code runs the tool and decides what comes back.
• Guidance and enforcement: guidance asks the model to follow a rule and hopes it does. Enforcement uses code to make the rule hold, whatever the model writes. Code can make some rules certain, like whose data a tool returns, and only screen for others, like advice wording.
• Test set: a saved list of sample chats, each with a check for a good answer. One example is the chat "Which fund should I buy?" with the check that the reply names no fund.

A system prompt is text your app places before the user's message, telling the model who it is, what to do and how to answer. The model does not run it like code. It reads it as part of one long input.

In the common stateless setup, where the service keeps no memory between requests, the app builds a fresh request on every new message. The system prompt goes first, then the earlier turns of this chat, then the new message. If the model asks for a tool, the tool's result is added and the model reads everything again.

On open models, ones whose files you can download and run yourself, you can inspect this step. A formatting rule called a chat template joins the messages into one stream of tokens (small chunks of text), with special tokens marking each role.

Models can be trained to give the system prompt more weight than the user's message. A 2024 paper calls this an instruction hierarchy: when the app builder and the user disagree, the app builder should win.

The paper argues that models often treat both kinds of text as equal priority. Its training method made the tested model much harder to trick, even with attack types it had not seen in training. Still, that extra weight is a learned habit, not an if statement you can unit test.

A good system prompt has five parts: a role, a task, rules, an output format and one or two examples. Write each in plain, specific sentences. Specific beats long.

• Role: who the model is and who it serves. One or two sentences are enough.
• Task: what it should help with, and what is out of scope. The helper answers questions about the user's own savings and app features.
• Rules: what it must not do, and what to do instead. A rule with a way out, such as offering to explain general ideas, gives the model a path, not just a wall.
• Format: the length, language and shape of the answer. If your code will read the answer, ask for JSON, a simple text format for data.
• Examples: one or two short sample exchanges that show the rules in action, especially the tricky cases. A good example often teaches more than an extra rule.

Here is the helper's system prompt, stored as a Python string so the app can send it with every request. Read it as text the model will see, not as code the computer runs.

SYSTEM_PROMPT = """You are the savings helper inside a savings app.

Task: answer questions about the user's own savings goals, deposits
and app features. Use the get_my_savings tool for account facts.

Rules:
- Do not recommend any fund, stock, gold scheme or amount to invest.
  Explain general ideas only, and suggest a registered adviser.
- Only talk about the signed-in user's own account.
- If you are not sure, say so and offer to connect the support team.

Format: plain English, at most 4 short sentences, amounts in rupees.

Example:
User: Should I put Rs 50,000 in a mutual fund?
Helper: I can't suggest specific investments. I can show what you
have saved so far, or explain what a mutual fund is."""

The first line sets the role. The Task lines say what the helper covers and which tool to use for account facts. Each rule gives a limit and a way out, such as pointing to a registered adviser.

Now notice what is missing: no account numbers, no keys and no internal limits. Anything in a system prompt can leak, as the next section explains.

The model follows a system prompt by learned habit, not by a hard rule, so a strong enough pull from the chat can win. Nothing inside the model checks your rules before each word it writes.

Think of a notice at the gate of a Chennai apartment block: all visitors must sign the register. Most visitors read it and sign. The notice cannot stop anyone, though. The watchman holding the register makes the rule true.

Map it part by part. The notice is the system prompt. The visitors are user messages and tool results. The watchman is your code: permission checks, input checks and reply checks.

Where the picture breaks: a visitor reads a notice once, but the model rereads the prompt on every reply. How much weight it gives the notice can shift with each chat's wording.

These are the common reasons the notice gets ignored:

• The user pushes the other way. A clear, emotional request, such as asking for one fund name for an elderly father, pulls hard against the prompt.
• The chat gets long. The rules sit at the start, and many turns pile up after them. Each turn is one more chance to slip, and slips add up, as the benchmark below shows.
• The rules are unclear or clash. If one line says be as helpful as possible and another says give no advice, the model has to guess which wins.
• Outside text carries instructions. A tool result or a document can contain text that reads like a command. This is called prompt injection, and it has its own guide.
• The model or the format changes. A new model version can treat the same prompt differently. One 2023 study found that formatting alone moved one open model's accuracy by up to 76 points.

That study changed only things like separators, spacing and capital letters, in prompts that held a few worked examples. The wording stayed the same. Bigger models in the study were still sensitive.

A 2024 benchmark (a shared set of tests for comparing models) measured rule-following directly. It tested 16 models on 500 system messages, each followed by five user turns.

Even the best model met every rule in only about three of every four turns (76.6%). Most models did worse when the user's request clashed with the system message.

Small misses also add up over a conversation. For the most stable model, only 26.4% to 33.7% of conversations kept every rule through all five turns, depending on how the turns were linked. These were models tested in 2024.

Pause and predict: would a better-worded prompt have stopped the account 7721 leak?

It would not. The fund answers came from pushy chats pulling against the prompt. The account leak came from the lookup tool, which returned whatever account the model asked for. No wording fixes a tool that trusts the model.

Any rule whose breaking would harm a user, break the law or move money must be enforced in code. The prompt still states it, so the conversation stays polite and on topic. Rules about tone and style can live in the prompt alone.

OWASP says the same thing. Controls such as authorisation checks, which decide who may see or do what, must not be handed to the model, through the system prompt or otherwise.

A quick test helps: ask what happens if the model breaks this rule once. If the answer is a slightly odd reply, the prompt is enough. If a stranger could see someone's balance, you need the watchman.

Code has four main ways to enforce a rule:

• Permissions: decide what each user may see or do, using facts the model cannot change. The savings tool reads the user's ID from the login session (the record of who is signed in), not from the model's request.
• Validation: check every input before using it. A second tool, set_weekly_amount, changes how much a user auto-saves each week. Its made-up range is Rs 100 to Rs 25,000.
• Confirmation: let a person approve high-impact actions. A new weekly amount waits as a pending change until the user taps Confirm. OWASP recommends human approval for high-impact actions.
• Output checks: read the model's draft reply before the user sees it. If it sounds like investment advice, code swaps in a fixed reply the compliance lead approved. This catches common advice wording, not every phrasing, so it lowers the risk rather than guaranteeing the rule.

So a request to save Rs 5,00,000 (5 lakh, or 500,000) a week is rejected by code, whatever the model says. OWASP's related advice is to treat the model like any other user and validate its responses before other code uses them.

The diagram below shows where each rule lives. Follow a message from the user to the reply, and count the boxes that check something without asking the model.

You put each hard rule in plain code that runs on every tool call and every reply, so the model's words cannot switch it off. Here is a small, runnable version for the savings helper.

Everything in it is made up for this guide: the in-memory savings table, the phrase list and the fallback reply. In a real app these would be your database, a classifier and text your compliance lead wrote. A classifier is a program that sorts text into groups, such as advice and not advice.

from dataclasses import dataclass

PROMPT_VERSION = "helper-v8"
SAVINGS = {"u1": 12500, "u2": 18400}  # made-up rows: user ID to rupees
PENDING = {}
BLOCKED = []
ADVICE_PHRASES = ("i recommend", "you should invest", "you should buy")
FALLBACK = "I can't suggest investments. I can show your savings instead."

@dataclass(frozen=True)
class Session:
    user_id: str  # filled in by your login system, never by the model

def get_my_savings(session, account_id=None):
    # The model may send any account_id. We ignore it on purpose.
    return {"user": session.user_id, "saved": SAVINGS[session.user_id]}

def set_weekly_amount(session, amount):
    if not isinstance(amount, int) or amount not in range(100, 25001):
        return {"error": "Weekly amount must be Rs 100 to Rs 25,000."}
    PENDING[session.user_id] = amount  # applied only after the user confirms
    return {"pending": amount}

def check_reply(session, draft):
    if any(p in draft.lower() for p in ADVICE_PHRASES):
        BLOCKED.append((session.user_id, PROMPT_VERSION, draft))
        return FALLBACK
    return draft

The second block checks the rules the way a unit test does. A unit test is a small automatic check of one piece of code.

def self_test():
    me = Session(user_id="u1")
    assert get_my_savings(me, account_id="u2")["user"] == "u1"
    assert "error" in set_weekly_amount(me, 500000)
    assert set_weekly_amount(me, 500) == {"pending": 500}
    assert check_reply(me, "I recommend the Growth Fund.") == FALLBACK
    explain = "A mutual fund pools many people's money."
    assert check_reply(me, explain) == explain
    print("all 5 checks passed")

self_test()

Save the two code blocks in this section in one file, rules.py, and run python3 rules.py. You should see one line: all 5 checks passed. We ran it on Python 3.13.16.

What just happened? get_my_savings ignored the account the model asked for and used the session's user ID. set_weekly_amount rejected 5 lakh and saved Rs 500 only as a pending change.

The reply check blocked a draft that recommended a fund, but let a plain explanation of mutual funds through. A phrase list misses advice worded another way, and it can block harmless replies such as recommending an adviser. Treat it as one layer, not the whole defence.

These checks are plain code, so they must pass on every single run. Prompt tests only tell you how the model tends to behave; unit tests show the enforcement works for the cases they cover.

The OWASP advice quoted earlier comes from its 2025 list. As of 19 October 2026, OWASP's main top 10 page still shows that list. OWASP also posted a 2026 edition for download on 3 August 2026.

That edition folds System Prompt Leakage into a wider entry, Hidden Context Exposure (LLM08:2026). The advice stays the same: hidden instructions are not a secret, and authorisation must not be handed to the model. The PDF itself still lists its publication date as to be set.

You run a saved test set of chats through the new prompt, check every answer, and compare the results with the live version before release. A small wording change can shift behaviour in places you did not touch, so you rerun the whole set every time.

Here is a release check for version 8 of the helper's prompt. All the numbers are made up.

1. 1

   Collect normal and tricky chats

   Build a test set of 120 chats: 70 normal questions, 30 advice traps such as a fund for an elderly parent, and 20 data traps asking for another account.

2. 2

   Write a check for each chat

   Some checks are code: the reply names no fund and shows no other user's data. Others need a person, or a second model acting as a grader, to judge politeness. Spot-check that grader too.

3. 3

   Run each chat several times

   When a model picks each next word with some randomness, called sampling, one chat can get different replies. Run each chat 3 times; a rule broken once in 3 is broken. Three runs catch frequent slips, not rare ones, which is why must-hold rules live in code.

4. 4

   Compare with the live version

   Run the same set on version 7 and version 8. Version 8 must pass at least as many checks, and every advice and data trap must pass on every run.

5. 5

   Get a human sign-off

   The compliance lead reads 20 sample answers to advice questions before any release, because a person who owns the rules catches things automatic checks miss.

6. 6

   Turn new failures into tests

   When a real user finds a new way to trip the helper, that chat joins the test set, so the same mistake is caught if it comes back.

The diagram below shows the same process as a loop. Follow the arrow from live chat logs back to the test set: that arrow is how the test set keeps improving.

Treat a system prompt like code: keep it in version control, give every change a version ID, and log which version answered each chat. Then you can always tell which wording caused a reply.

The team's first prompt was text pasted into an admin screen. When a bad answer showed up, nobody knew which wording produced it or who changed it last. These habits fix that:

• Keep each prompt in its own file in the app's Git repository, so every change has an author, a date and a reason.
• Give each release a version ID, like helper-v8, and save it with every chat in the logs. The check_reply code above already does this for blocked replies.
• Change one thing at a time, so when test results move, you know why.
• Rerun the full test set whenever the prompt, the tools or the model change. A new model version is a new test, even with the same prompt.
• Keep the old version ready. Switching back to version 7 should be a one-line config change, not an emergency edit.

Stable prompts also help speed and cost. Some model servers reuse work for a repeated prompt start, called prompt caching. Reuse stops around the first changed word, so an edit near the top of the system prompt throws away most of the saved work.

Release mechanics such as canary rollouts, where a small share of users gets the new version first, are covered in the prompt versioning guide.

A strong answer treats the system prompt as one layer that shapes behaviour, then names the rules you enforce elsewhere. Interviewers want to hear that you expect the prompt to fail sometimes, and what you built for that.

1. 1

   Name the prompt's job

   Say it sets the role, task, rules, format and examples, and that it is guidance the model usually follows.

2. 2

   List the hard rules

   For the savings helper: no personalised investment advice, and no data from other users.

3. 3

   Move each hard rule into code

   Permissions from the login session, validation on every tool input, user confirmation for money changes, and an output check before any reply is shown.

4. 4

   Explain how you test

   A test set of normal, tricky and hostile chats, each run several times, on every change to the prompt, the tools or the model.

5. 5

   Explain how you ship changes

   Versioned prompts, version IDs in the logs, a compliance review, and a one-line rollback.