Copilot Studio Kit: Testing and Governance Tools Explained

Copilot Studio Kit is a set of tools around Microsoft Copilot Studio for testing, analyzing, and governing custom assistants; Microsoft now documents it as Copilot Agent Kit. It helps teams collect evidence about an assistant’s behavior and configuration, rather than treating a convincing demonstration as proof that it is ready for work.

fenbs can keep the rollout tasks, findings, and human release decisions together while the kit performs its Microsoft-specific work. You still need an owner who chooses what to test, understands the results, and approves the next step. If you are new to the builder, start with the Copilot Studio overview.

What is the kit, and why do its names differ?

The Power Customer Advisory Team, or Power CAT, maintains the toolkit. Microsoft’s current overview↗ (opens in a new tab) calls it Copilot Agent Kit, while the source repository address still contains Power-CAT-Copilot-Studio-Kit. Search results, older screenshots, and installed solutions may therefore use different names.

Think of the kit as additional applications and components you deploy and configure for a defined purpose. It is not a separate conversational model and does not remove the need to build the assistant itself. Before following a tutorial, compare its component names with the release you intend to install. A procedure for an older package may omit newer dependencies or controls.

The practical question is not whether to install everything because it exists. Ask what uncertainty blocks your next decision: unreliable answers, unknown ownership, unclear configuration, or insufficient review evidence. That answer should determine the first component you evaluate.

Which components answer which questions?

The component catalog covers more than test automation. These groups summarize capabilities listed in Microsoft’s overview; availability and setup should be checked for the release you use.

Microsoft’s source repository↗ (opens in a new tab) also lists capabilities such as Agent Change Tracker and Component Library with preview labels. Treat preview status as a reason to inspect the release notes and suitability for your environment. The catalog is a navigation aid, not evidence that every component is configured or appropriate for your production service.

What can automated testing actually establish?

Microsoft’s testing documentation↗ (opens in a new tab) describes response matching, attachment checks, topic matching, generative-answer evaluation, and multi-turn tests. It also describes plan validation, which checks whether an assistant’s generated plan includes expected tools. Some checks need additional services such as Dataverse, AI Builder, or Application Insights.

These test types answer different questions. An exact response match suits a fixed acknowledgment. A multi-turn test can check a conversation that collects details before taking a step. Generative answers need criteria that allow valid wording changes while rejecting missing facts or invented commitments. A tool appearing in a plan does not establish that the downstream operation completed correctly.

For example, a facilities assistant should explain an approved maintenance process without inventing a promised repair date. Write a test that identifies the required facts and prohibited claim. Then include a case with missing information, a confusing follow-up, and a request outside its purpose. Review a sample manually to check whether automated grading agrees with the actual requirement.

How do configuration checks differ from behavior tests?

The Agent Review Tool↗ (opens in a new tab) analyzes assistant solutions and reports potential implementation problems with recommendations. A configuration finding can reveal a risk that ordinary happy-path conversations miss. Conversely, a clean configuration review cannot establish that answers are complete, correct, or useful for every user.

Compliance Hub adds a different workflow. Microsoft describes it as evaluating configured controls using inventory information, creating cases when thresholds are breached, and supporting review or enforcement actions. Its documentation↗ (opens in a new tab) explains dependencies and actions, including quarantine or deletion. Understand those consequences before enabling enforcement.

Assign an owner to each kind of finding. A missing owner, an unsuitable connection, and an inaccurate answer need different fixes. Record whether a finding was corrected, accepted by an authorized person, or remains open. Keep the reason in fenbs Decisions and rules when it changes what the team is permitted to do.

What should you check before installation?

Read the current prerequisites↗ (opens in a new tab) before importing a solution. Microsoft lists a Power Platform environment with Dataverse, installation privileges, appropriate licensing, and Creator Kit dependencies. Some capabilities consume additional capacity or require telemetry configuration. Check the vendor’s requirements instead of assuming that access to Copilot Studio covers the whole installation.

Connector permissions deserve specific review. Microsoft states that the target environment’s data policies must allow the connectors shipped in the solution, even if you do not intend to use every feature afterward. Ask an administrator to inspect that requirement against your restrictions. Do not broaden a policy merely to make an import succeed without understanding the resulting access.

• Choose a controlled evaluation environment and identify who can change its policies.
• Use sample or suitably approved test information and define who may read stored results.
• Check connections, dependencies, authentication support, and release notes for the exact package.
• Name an owner for updates, failed runs, storage, and cleanup after the evaluation.

How should you evaluate the kit on one assistant?

Start with one assistant and a short list of decisions the evaluation must support. For example: can it answer handbook questions, refuse unrelated requests, and avoid claims that require a manager’s approval? Pick the smallest component set that produces evidence for those questions, then save a baseline before changing the assistant.

Repeat the same checks after a deliberate change and inspect the difference. Include failures as useful results. If the test cannot distinguish a harmless wording change from a serious factual error, improve the criteria before treating its pass rate as a release gate. Broader Microsoft Copilot agents can have different purposes, so avoid borrowing another team’s test set unchanged.

How to do this with fenbs

1. Create a free account and add a project for your kit evaluation, with a clear description of the assistant and intended audience.
2. Record the acceptance criteria a person approved in Decisions and rules. Include required answers, prohibited actions, and the reviewer responsible for release.
3. Add tasks for installation review, test cases, and each finding. Put the approach in the plan and record actual checks and limitations in the task’s testing fields.
4. Use Where we left off to save the tested version, remaining questions, and next step. Ask a connected assistant to read project context before continuing the evaluation.

fenbs does not install the kit, execute its tests, or receive Microsoft telemetry automatically. It is not the right fit when you need a test runner or Microsoft administration dashboard. Its role is retaining the work and the decisions people make from that evidence.

What should remain a human decision?

A tool can identify a failed check without deciding whether the remaining risk is acceptable. The release owner must consider the intended users, consequences, unresolved findings, and recovery procedure. A department FAQ assistant and an assistant that changes business records need different acceptance conditions.

Keep instructions readable using Copilot instruction examples, and use the AI governance hub for the broader policy context. A useful evaluation ends with an explicit result: proceed within the approved scope, fix named problems, or stop until a person resolves the outstanding question.