Test Management in Jira: Xray, Zephyr and Lighter Options

Jira tracks work, not test runs, so test management in Jira means adding an app or adopting a convention. The two best-known apps take different shapes. Xray for Jira makes tests ordinary Jira work items (Test, Precondition, Test Set, Test Plan, Test Execution) and records each result as a test run. Zephyr keeps test cases in its own library of folders inside Jira and links them to your stories and bugs. The lighter options are a Test work type you add yourself, or a separate test tool that links into Jira. Which fits depends on how many tests you rerun, and how much you need to prove afterward.

What a Jira test management tool adds

Out of the box, a Jira work item can say “test the checkout” and be moved to Done once. Real testing needs more than that, and these five things are what the apps add:

• Reusable test cases: steps, data and expected results you can run again next release.
• Executions: one run of a set of tests against one build or environment, with a result per test.
• History: how a test did last time, and the time before.
• Coverage: which stories and bugs have tests, and whether those tests passed.
• Automation results: pass and fail imported from your CI, next to the manual results.

If you only need the first item, you may not need an app at all. If you need the last three, you almost certainly do.

Xray for Jira: tests as work items

Xray describes itself as “a complete Test Management tool for Jira”. The defining choice, set out in About Xray↗ (opens in a new tab), is that its main entities are Jira work types: Test, Precondition, Test Set, Test Plan, Test Execution and Sub-Test Execution. A test has a key like any other work item, sits in your spaces, follows a workflow, and shows up in JQL, boards and dashboards.

Results live one level down. Xray’s page on the Test Execution↗ (opens in a new tab) defines it as a work type that “aggregates a user-determined collection of Tests”, and the pairing of one execution with one test is a test run, with statuses such as To Do, Executing, Passed and Failed. A Test Plan groups tests for a release and the executions that ran them, and from a plan you can create a new execution for only the tests that are still failing.

• Test types: manual step-by-step tests, Cucumber and other BDD scenarios, and generic automated tests.
• Automation: results from JUnit, TestNG, NUnit, Robot Framework and other runners come in through Xray’s REST API, so a CI job can post results to an execution.
• Editions: Xray Cloud for Jira Cloud, and a Server and Data Center edition with its own documentation.

The trade-off follows from the design. Because tests are work items, your Jira space fills with them, and filters, boards and permissions have to allow for that. Teams usually keep tests in their own space or exclude the test work types from the development board.

Zephyr: a test library inside Jira

First, the name. SmartBear says in its Zephyr documentation↗ (opens in a new tab) that it has “consolidated Zephyr Squad and Zephyr Scale into a single product family: Zephyr”, offered in editions it names Essential, Standard and Advanced. Older guides that compare Squad with Scale describe products that are now one family, so check which documentation your site actually uses.

Zephyr organizes testing in three layers. Test cases “define the steps, data, and expected results”; test cycles group tests for execution “within a release or sprint”; test plans combine cycles across a larger release. The key difference from Xray is where a test case lives. SmartBear’s guide to creating a test case↗ (opens in a new tab) puts it in Zephyr’s own library, filed in folders, with a name (the only required field), a status such as Draft or Approved, and a Test Script tab for the steps. You then link test cases and cycles to any Jira work item for traceability.

• BDD: Gherkin scenarios are a supported test case format.
• Automation and CI: SmartBear lists integrations with tools such as Jenkins, Bamboo, Cypress and JUnit.
• AI: the documentation describes AI features for drafting tests and analyzing coverage.

The trade-off is the mirror image of Xray’s. Your Jira spaces stay free of test work items, but tests are not ordinary work items, so you work with them through Zephyr’s screens and links rather than plain JQL.

Xray vs Zephyr, in five lines

• Where a test lives: Xray, as a Jira work item; Zephyr, in its own folder library linked to work items.
• Grouping: Xray, Test Sets and Test Plans; Zephyr, test cycles and test plans.
• Results: Xray, test runs inside a Test Execution; Zephyr, executions inside a test cycle.
• Searching: Xray tests answer to JQL like any work item; Zephyr test cases are found in Zephyr’s library and through their links to work items.
• Pick Xray if you want tests in the same lists, permissions and reports as everything else. Pick Zephyr if you want testing kept out of the development spaces.

Lighter option 1: test cases as work items, no app

A small team that runs twenty manual checks before each release can do it with Jira alone. A Jira admin can add a work type↗ (opens in a new tab) called Test Case and associate it with the space through a work type scheme. Each test case is a work item, written with the same care as any Jira ticket, with the steps in the description and a link to the story it covers. Each release, you clone the set, run it, and record the result.

Title: TC-Checkout-07: ZIP+4 accepted at shipping
Type: Test Case    Labels: regression, checkout
Links: tests PAY-142

Preconditions
Signed-in customer, one item in the cart.

Steps
1. Go to checkout, enter a US shipping address.
2. Enter ZIP 94105-1804. Select Continue.
3. Enter ZIP 9410. Select Continue.

Expected
Step 2: shipping options appear, sales tax is shown.
Step 3: "Enter a valid ZIP code", order blocked.

Result for release 4.2 (comment, then close)
Passed in Chrome on macOS and Safari on iPhone. Build 4.2.0-rc1.

What you lose: there are no test runs, so history is a trail of comments and closed clones, and there is no coverage report. When the cloning becomes a chore, that is the signal to try an app. For writing the steps themselves, use our test case template, and file every failure as a bug linked to the test.

Lighter option 2: a test tool beside Jira

Some teams keep test cases in a dedicated test management tool and connect it to Jira rather than installing an app inside it. TestRail is a common example. Its support guide to reference and defect integrations describes two halves: a References field on test cases, runs and plans that links the stories being tested, and a defect integration that reports bugs into your tracker straight from a test result. Testing stays in one tool, the defects land in the other, and each side keeps its own permissions and reports.

How to choose

• Fewer than about fifty manual checks, run before each release: a Test Case work type and a checklist. Revisit when it hurts.
• You need coverage reports, history per test and CI results in Jira: Xray or Zephyr. Try both against the same twenty tests.
• Tests should be searchable and reportable like any other work item: lean toward Xray.
• Development spaces should stay free of test items: lean toward Zephyr, or a separate tool linked into Jira.
• Auditors or customers ask what was tested in each release: pick the app whose execution report answers that question on one screen.

If what you need is “was this checked?”

Plenty of small teams do not run test suites at all. They need to know, for each change, whether someone checked it and how. That is a field on the task, not a product. On a fenbs board every task carries a test status (Not tested, Tested, Partly tested, Failed, or Needs owner check for what only a person can confirm, such as a store build or a real payment) and test notes saying what was checked, where and with what result. An AI assistant working the board over MCP fills in the same two fields when it finishes, so “tested” means the same thing whoever did the work.

Be clear that this is not test management. fenbs has no test case library, no test runs, no execution history and no coverage report. Use it beside a QA checklist or a regression testing checklist; if you need runs and coverage, use Xray or Zephyr.