How to Add AI to Your App: Scope, Context, and Testing

Adding AI to an application is a product decision before it is an engineering task. The apps that add AI well start with one specific problem their users have: drafting a reply, summarizing a document, extracting data from a form. Apps that add AI poorly start with "how do we add AI?" and end up with a feature nobody uses.

Once you have the problem, the engineering pattern is consistent: choose an AI provider, supply context through the API, constrain what the AI can do, and evaluate whether it is solving the problem. Tools like fenbs keep the product requirements and the development decisions in one place so every developer working on the feature starts from the same brief.

Step 1: Choose a focused feature

The most successful AI features in applications do one thing well. A legal document reviewer that highlights clauses a lawyer should check. An email composer that drafts a reply from bullet points. A form filler that extracts fields from an uploaded document. Each one is narrow enough to evaluate and explain to users.

Avoid adding a general chat interface as a first step unless the product's core value is conversation. A general chat feature is hard to evaluate and easy to misuse. A focused feature has a clear success criterion: did the AI extract the right fields? Did the draft need fewer edits than before?

Step 2: Choose an AI provider

The main options for most application developers are the Anthropic API (Claude models), the OpenAI API (GPT models), and Google's Gemini API. All three provide a REST API with a messages format, tool use, and streaming. Choose based on the capabilities your feature needs, the pricing model that fits your usage, and the terms that match your data handling requirements.

• Test your specific task on each provider before committing. Performance varies by task type.
• Read the data handling and retention policies before sending user data to any API.
• Check the terms for your industry if your application handles sensitive data.
• Start with the simplest integration: a single API call before adding tool use or streaming.

See what is Cursor AI for how AI APIs are used in a developer tool, and google AI tools for an overview of the Gemini API options.

Step 3: Supply context through the system prompt

The system prompt is the instruction you send with every API call. It tells the model what the application does, who the user is, and how it should behave. A well-written system prompt makes the AI behave consistently across sessions and reduces the need for users to explain context they have already provided.

• Describe the application and the user's role.
• State the format the AI should return: JSON, markdown, plain text.
• List what the AI must not do: mention competitors, make promises, suggest actions outside the app.
• Include any business rules that apply to the task.

Update the system prompt when the application changes or when testing reveals consistent errors. Treat it as a piece of the application that needs maintenance.

Step 4: Define action boundaries

If the AI feature can take actions — send a message, update a record, call an API — define the boundaries before building. What can it do without confirmation? What requires a user to approve? What is completely off limits?

A feature that drafts text for a human to send is lower risk than one that sends automatically. A feature that reads data is lower risk than one that writes. Build the more constrained version first: it is safer to expand later than to retract after users depend on the wider behavior.

• List every action the AI feature can trigger.
• Mark each as read-only, draft-only, or write.
• Require user confirmation for any write action that is hard to reverse.
• Log every action the AI takes with a timestamp and the input that triggered it.

Step 5: Evaluate before shipping

Before users see the feature, test it on real examples from your domain. A tone analyzer that works on sample text may fail on real user-submitted content. Build a small test set of inputs and expected outputs, run the feature against it, and score the results.

1. Collect 20-50 real examples of the input the feature will receive.
2. Define what a correct output looks like for each example.
3. Run the feature and score the outputs.
4. Identify the failure modes: what types of input produce wrong results?
5. Improve the system prompt or the feature scope to address the failures, then retest.

How to do this with fenbs

1. Create a free account and add a project for the application.
2. Create a task for each AI feature: what it does, the user problem it solves, the success criterion.
3. Write the system prompt and the action boundary decisions in Decisions and rules.
4. Connect your coding assistant to fenbs from Integrations. It reads the feature brief before each development session.
5. After testing, record the test results in the task notes and note what still needs work before shipping.