AI Research Assistant Project: Scope, Sources, and Evaluation

An AI research assistant project succeeds when you can point to a specific question it answers well, with citations you can verify, results that are consistent across runs, and a clear statement of what it cannot do. Scope and evaluation criteria come before implementation.

The most common failure mode is building retrieval and generation before deciding how to measure accuracy. Without a test set, you have no way to know whether a change improved the assistant or just changed it. Tools like fenbs keep the project brief, the evaluation decisions, and the verified source links in one place so they survive across development sessions.

Define the scope before writing code

A research assistant with a narrow scope is faster to build and easier to evaluate than a general one. Start by answering three questions: what specific questions should the assistant answer well, what documents or sources will it read, and what does a good answer look like?

Say a three-person law firm wants an assistant that answers questions about OSHA regulations from the official OSHA website. That scope is narrow enough to build in a weekend and specific enough to test with real questions. A "general research assistant that answers anything" is not a project scope; it is a description of every search engine ever built.

Choosing sources and building the document set

The quality of the document set determines the quality of the assistant. A retrieval assistant can only return what is in the documents it has indexed. Include primary sources (the official document, the original paper, the vendor's own help page) rather than secondary summaries, which may paraphrase incorrectly or omit important qualifications.

Record each source in your project notes: the URL, the date you retrieved it, and whether it is a primary or secondary source. These records matter when a citation is challenged and you need to find the original. For a law firm, that record may also be required for professional responsibility reasons.

Implementing retrieval and citation

A basic retrieval pipeline has four steps: chunk the documents into passages, embed each passage into a vector, store the vectors in a database, and at query time embed the question and retrieve the most similar passages. Pass the retrieved passages to the LLM as context and ask it to answer using only what is in the passages.

Citation tracking means recording which passage each part of the answer came from. The simplest approach: include a source label in each passage before embedding, and require the LLM to include the source label whenever it uses information from a passage. After generation, map each label back to the original document and URL. Test citation accuracy manually on your first ten answers before trusting it at scale.

Building an evaluation set

An evaluation set is a list of questions with known correct answers drawn from your document set. Build it before you build the assistant. Twenty to fifty questions is enough for a prototype. For each question, note the correct answer, the document it comes from, and the passage that contains it.

After building the assistant, run every evaluation question through it. Measure three things: whether the answer is factually correct, whether the citation points to the right document, and whether the answer is complete (it includes the full correct answer, not just part of it). A pass rate below 70 percent on a narrow-scope assistant usually signals a retrieval problem, not a generation problem.

Common retrieval problems and how to diagnose them

When the assistant gives wrong answers, the cause is usually one of three things: the right passage was not in the document set, the right passage was not retrieved (ranking failure), or the right passage was retrieved but the LLM did not use it correctly (generation failure). Diagnose each type separately.

• Missing document: search your document set for the answer manually. If you cannot find it, the document is not indexed.
• Ranking failure: retrieve the top ten passages for the failing question and check whether the correct passage is in the list. If it is in position 8 or 9, increase k or improve chunking.
• Generation failure: pass the correct passage directly to the LLM as context and ask the question again. If it answers correctly, the retrieval is the problem. If it still answers incorrectly, the prompt or the LLM is the problem.

How to do this with fenbs

Use fenbs to keep the project organized across the sessions and team members who contribute to building and evaluating the assistant.

1. Create a free account and create a project for the research assistant build.
2. Write the scope as the project description: the questions it should answer, the sources it reads, and what a good answer looks like.
3. Add the evaluation decisions as Decisions and rules: the pass rate threshold, the citation accuracy standard, the sources that are in scope.
4. Track the source collection as tasks: one task per source type, with the verified URLs linked in the task notes.
5. Connect your assistant from /integrations so it reads the project context at the start of each development session.