Context Window Test: Check an AI on Your Own Work

A useful context window test checks whether an AI can find the right facts, follow the right rules, and admit what your supplied material does not establish. Build it from representative work, write the expected answers first, and repeat it with controlled changes to input length and information placement.

The context window is the information available for the model’s current response; tokens measure the pieces it processes. fenbs can keep your test prompts, project decisions, and lessons together so another assistant or teammate can run the same evaluation later.

How do you set up a useful first test?

1. Choose one real workflow, such as checking a design proposal against approved requirements. Use public, synthetic, or appropriately approved material.
2. Freeze a small source packet and give every document a clear name and version. Keep an unchanged copy for later runs.
3. Write questions and expected answers before testing. Include the supporting passage and the mistakes that would make an answer unacceptable.
4. Run the test in a fresh session using the same model, settings, source packet, and prompt for each comparison.
5. Record the full answers, score them against the key, and rerun the cases before drawing conclusions.

Begin with a packet a person can review completely. An enormous collection makes it difficult to tell whether an answer is wrong or the test itself is ambiguous. Your first goal is a dependable check of a narrow task, not a leaderboard for every possible use of AI.

A test plan template helps define the scope and acceptance criteria. Keep those criteria stable during a comparison; changing the key after seeing a preferred model’s answer makes the result hard to trust.

Which questions should the source packet support?

Use several kinds of question because finding a sentence is different from applying it. The RULER research↗ (opens in a new tab) extends simple retrieval tests with tasks involving multiple facts, tracing, and aggregation. Borrow the principle of testing different behaviors without treating your small exercise as an official benchmark.

For example, create a fictional proposal packet with three clearly labeled documents. The approved requirement says account creation must remain optional. A later design note proposes requiring an account, but nobody has approved it. A schedule says the next review is on October 28, without saying who will attend.

These are invented test materials, not a customer story or measured result. Add exact source labels and passages to your private answer key. A test case template can give each question its own expected outcome and failure criteria.

How should you score the answers?

Score the parts that matter to the workflow separately. Did the assistant identify the correct fact? Did it use the approved source? Did it preserve the restriction? Did it avoid inventing an attendee? A polished paragraph should not receive full credit if it reverses the rule.

• Correctness: the answer matches the written key.
• Evidence: the named document and supporting passage actually justify the answer.
• Instruction following: the answer respects approval status and task limits.
• Uncertainty: missing information stays missing instead of becoming a confident guess.
• Completeness: the answer covers every required part without adding an unsupported action.

Mark critical failures explicitly. Treating a proposal as permission might make a coding session unacceptable even if every date was recalled correctly. Preserve the individual results alongside any total score so that a high average cannot hide a failure that matters to your work.

Use a second reviewer for ambiguous cases when possible. If two people cannot agree on what the packet establishes, improve the question or source before comparing models. The test should measure the assistant’s behavior rather than your ability to interpret an unclear assignment.

How do you vary length and information placement?

Start with a short version, then add relevant supporting material in controlled stages. Record the actual input token count when your interface provides it. Change one factor at a time so you can distinguish a length problem from a changed question or a newly conflicting source.

Also move the decisive passage between the beginning, middle, and end while keeping its wording and authority unchanged. Lost in the Middle↗ (opens in a new tab) studied sensitivity to information position in long inputs. Your own test can check whether placement changes the outcome for your selected model and task.

Repeat runs at each setting, preserving failures as well as successes. Do not call the largest successful input a universal safe limit. It describes performance on that packet, under those settings, for those questions; a different task may behave differently.

If you need to interpret published evaluation methods, see context window benchmarks. The AI context window explainer covers capacity itself; this exercise measures whether your workflow uses that capacity well.

What can distort the result?

A continuing conversation can reveal earlier answers to later runs. New browsing results can change the evidence, and application summaries can change the input without you noticing. Use fresh sessions, note which tools were enabled, and distinguish a model-only test from a test of an entire application.

Do not ask the same model to be the only judge of its own answer. Automatic checks can help with exact identifiers and required fields, but a person should review whether the cited evidence supports the conclusion. Keep incorrect citations visible rather than scoring only the final sentence.

When the procedure stabilizes, test automation can help rerun fixed cases. Automation should preserve the packet version, prompt, settings, and raw outputs so a future reviewer can explain a changed result.

What should you save after each run?

Record the model identifier, application, date, settings, source version, question set, token count if known, and tool permissions. Save each answer and its score with the reason for any failure. If an application hides part of the input, record that limitation instead of pretending the comparison is fully controlled.

Turn an observed failure into a specific lesson: what happened, what it means for this workflow, and what to do differently. “The assistant treated a proposed change as approved; label proposals separately and check approval before editing” gives the next session something useful to apply.

How to do this with fenbs

1. Create a free account. It is free for one person, with no credit card.
2. Add an evaluation project and a task whose plan names the packet, questions, scoring rules, and acceptance criteria.
3. Save the reusable instructions in Skills and prompts, and keep source references in context notes or Links and listings.
4. Use a supported app from Integrations to read the task and carry out the evaluation. Record test results and limitations on the task.
5. Add a lesson for a meaningful failure and save Where we left off, including the next comparison to run.

fenbs does not run model benchmarks or grade responses automatically. Use an evaluation runner when you need that execution layer; fenbs fits when you need the project record, reusable prompts, and lessons to survive a change of assistant. Explore related workflows in the context engineering topic hub.