Jenkins and Bamboo Integration

qTest Manager allows for out-of-the-box integration with Jenkins and Bamboo. The integration collects executed unit test results while building projects in your CI server and reports them to qTest Manager. Our Plug-in does not execute your unit tests. It will only collect and submit test results (if any) to qTest.

To set up this feature:

  1. Activate the CI Integration feature in your qTest project.

  2. Map the CI execution status with corresponding qTest values.

  3. Configure the Test Run and Test Case organization.

  4. Install the qTest Plug-in on Jenkins or Bamboo and configure it to connect to your qTest project.

Supported Testing Frameworks

We support testing frameworks that are capable of generating JUnit XML format test reports.

  • For Jenkins, if your testing framework does not generate JUnit XML format test reports, you can utilize xUnit Plug-in to generate and publish test results before qTest Plug-in collects and reports them to qTest.

  • For Bamboo, the qTest Bamboo Plug-in does not support Bamboo Specs, and only JUnit XML format test reports are supported.

Supported Build Tools

  • Maven

  • Ant

  • Gradle

Activating the CI Integration

To receive Test Logs submitted from your CI server and associate them with the builds, you will need to enable this feature in your qTest project. This will need to be done for each project individually.

To activate the CI Integration:

  1. Select the appropriate qTest Project.

  2. Hover over the Gear icon and select Automation Settings from the drop-down menu.

  3. Switch on the Activation Status in the CI Tool Integration section.

  4. Map the automation execution statuses of your CI tools with the corresponding qTest values.

Mapping the CI Status

Activating the CI Tool Integration will also activate the Automation Integration feature. Therefore, it is required that execution statuses for your CI tools are mapped to the corresponding qTest values. The CI tools' default execution statuses are:

Jenkins:

  • PASSED

  • FAILED

  • SKIPPED

  • FIXED

  • REGRESSION

Bamboo:

  • SUCCESS

  • FAILED

  • SKIPPED

To map the CI statuses:

  1. Select the Add Mapping button.

  2. Enter the CI tools status and the corresponding qTest value.

  3. Select Save.

Organizing Test Runs

While setting up the CI Tool Integration, you have different options to manage your Test Runs and Test Cases.

  1. In one Test Suite per selected Release: One Test Suite is created under each selected Release on qTest Plug-in's configuration in your CI server. This is the default option.

    • Jenkins: Test Suites are named as [Jenkins Project Name] [mm-dd-yyyy], where [mm-dd-yyyy] is the suite's created date.

    • Bamboo: Test Suites are named as [Bamboo Plan Key] [mm-dd-yyyy], where [mm-dd-yyyy] is the suite's created date.

  2. In one Test Suite for all Releases: Test Runs are created under only one Test Suite regardless of selected release in your CI server's qTest plug-in configuration.

    • Jenkins: Test Suites are named as [Jenkins Project Name].

    • Bamboo: Test Suites are named as [Bamboo Plan Key].

  3. Test Case submitted from your CI server are located under a Module named [Jenkins Project Name] or [Bamboo Plan Key]. You can optionally select a parent module to contain the CI module. If no parent module is selected, the CI module is created in the root module.

Jenkins Integration

The Jenkins integration is available in two configurations:

  • Freestyle: Use qTest Jenkins Plug-in in your post-build actions to submit test results to your qTest Manager project.

  • Pipeline: Generate pipeline script from qTest Jenkins Plug-in and put it in your pipeline code as a step to submits test results to a qTest Manager project.

You have the option to use either type of Jenkins Integration to submit your test results to qTest Manager.

Install the qTest Jenkins Plug-in

This installation is required to use Jenkins Freestyle or Jenkins Pipeline.

  1. From the Jenkins Plug-in Manager, search for the qTest Plug-in and install.

  2. Add Submit JUnit Test Results to qTest as a post-build action.

    If your testing framework does not generate JUnit XML format test reports, you can use the xUnit Plug-in to generate and publish test results before the qTest Plug-in collects and reports them to qTest. Once the xUnit Plug-in is installed, add Publish xUnit test result report as a post-build action and then add Submit JUnit Test Results to qTest.

Notes

  • If you use xUnit Plug-in to generate and publish test results, make sure that the option Delete temporary JUnit files is cleared.

  • Maven projects: It is not required that there is a prior task to publish test results to Jenkins.

  • Freestyle and Multi-configuration projects: It is required that there is a prior task to publish test results to Jenkins. If not, no test results will be submitted to qTest and the Submission Status will show the status as SKIPPED for the builds.

Below is a screenshot of the qTest Jenkins Plug-in after being added as a post-build action.

Configure the qTest Jenkins Plug-in for Freestyle

  1. In qTest Manager, locate the Download qTest Resources Page  icon to open the page.

  2. Select the CI Tool Integration.

  3. Copy the API token from the Integration with Jenkins section. This allows Jenkins to access your qTest Manager instance.

    The API token is at a user level, which means it is different between qTest users.

  4. Navigate back to the Jenkins Plug-in Configuration page.

Configure the Jenkins Plug-in for Freestyle to Execute Tricentis Tosca Tests

  1. Select the Execute Tricentis Tosca Tests check box. Then enter the information below:

    • Path to Tricentis Tosca CI Executable: Enter the path to the Tosca CI client tool, which is either ToscaCIClient.exe or ToscaCIClient.rar, depending on the operating system this tool will execute.

    • Tricentis Tosca Command Line Arguments: Command line arguments to execute Tosca test.

    • Path to Results: Enter the path to the results generated by the Tosca CI client tool when it finished execution.

    The following screenshot shows an example configuration.

  2. Enter the following qTest Manager information:

    • qTest URL: The URL to access your qTest instance, such as https://demo.qtestnet.com.

    • API Key: This is the integration with Jenkins token you obtained from the qTest Manager Resources page.

    • Select the Retrieve Data button to load data from qTest Manager.

    • Select the qTest project that the build is integrating with and where you want to submit test results.

    The configuration should look like the following example:

Configure Options to Submit Test Results to Manager

  1. Configure the options to submit your Test Results to qTest Manager.

    • Submit test result to a Release as settings in your qTest Manager. By selecting this option, the qTest Plug-in will submit the test results to a Release in qTest Manager (refer to the Organizing Test Runs section for more information on setting this up). Select a Release, which will be the target release of submitted Test Runs.

    • Submit Test Results to an Existing Container. By selecting this option, you allow qTest Jenkins Plug-in to submit test results to a specific container in qTest Manager.

      • Container: Select a container where you want your test results to be submitted, such as a Release, a Test Cycle, or a Test Suite. The Plug-in will create a Test Suite directly under the selected container. Then it creates Test Runs underneath that Test Suite. It will then submit Test Logs to those Test Runs. If the Test Suite already exists, the Plug-in will not create duplicate Test Suites but will submit test results to that Test Suite.

      • Check this box to create a new test suite daily. If unchecked, a single test suite, with this job’s name, will be used: qTest Jenkins Plug-in will create a new Test Suite daily, then will create Test Runs underneath that Test Suite, and then will submit Test Logs to those Test Runs. If the Test Suite already exists (for example, if it was created by previous Jenkins built in the same day), then the Plug-in does not create a duplicate Test Suite but will submit test results to that Test Suite.

  2. (Optional) Environment: The selected Environment will be set as submitted Test Run's property.

    The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the newly selected Environment. The user must delete the existing Test Runs and create a new ones if the user wants to inherit the Environment from the Test Suite.

  3. Overwrite existing test steps: Selected by default. When test results are submitted to qTest Manager, the Test Case steps are also updated in the log. If the original Test Case contained test step details with manual test steps, these details would get overwritten, since the test step logs would create newer Test Case versions.

    If you wish to retain test step details from the existing test case, you can clear this option.

  4. Parse test results from testing tools: Selected by default. If selected, the Plug-in will scan for the XML files that contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the Plug-in will scan the whole project.

    This option is not available if you selected to execute Tosca tests.

  5. Utilize test results from the CI tool: In case there is one of the prior tasks that have already scanned and published test results to Jenkins, for example by using xUnit Plug-in, qTest Jenkins Plug-in will use the test results output from that task without scanning again.

    This option is not available if you selected to execute Tosca tests.

  6. Each JUnit Test Suite (class) = a qTest Test Case: Selected by default. A qTest Test Case is created from a JUnit Test Suite (class).

    This option is not available if you selected to execute Tosca tests.

  7. Each JUnit Test Case (method) = a qTest Test Case: A qTest Test Case is created from a JUnit Test Case (method).

    This option is not available if you selected to execute Tosca tests.

  8. Select Save to complete the configuration.

Generate Pipeline Script from qTest Jenkins Plug-in

If your Jenkins project uses the Jenkins Pipeline feature for advanced continuous integration scenarios, you can generate a pipeline script from the qTest Jenkins Plug-in to submit test results to qTest Manager. The script, once generated, can be used as a step in your entire pipeline scripts.

Prerequisites

Make sure your Jenkins system already has Jenkins Pipeline setup, which requires:

Setup Pipeline Syntax

  1. From your Jenkins project, select Pipeline Syntax in the left panel.

  2. The Pipeline Syntax page loads.

  3. Select Snippet Generator.

  4. In the Steps section, select submitJUnitTestResultsToqTest: Submit jUnit test result to qTest from the Sample Step list.

Configure qTest Jenkins Plug-in to Generate Pipeline Script

  1. Navigate to qTest Manager and locate the Download qTest Resources Page  icon to open the page.

  2. Select the CI Tool Integration section to open.

  3. Copy the API token from the Integration with Jenkins section. This allows Jenkins to access your qTest Manager instance.

    The API token is at a user level, which means it is different between qTest users.

  4. Navigate back to the Jenkins Pipeline Syntax page.

Configure the Jenkins Plug-in for Pipeline to Execute Tricentis Tosca Tests

  1. If you want the script to execute Tricentis Tosca tests, select the Execute Tricentis Tosca Tests check box. Then enter the following information:

    • Path to Tricentis Tosca CI Executable: Enter path to the Tosca CI client tool, which is either ToscaCIClient.exe or ToscaCIClient.rar, depending on the operating system this tool is going to execute.

    • Tricentis Tosca Command Line Arguments: Command line arguments to execute Tosca test.

    • Path to Results: Enter path to the results generated by the Tosca CI client tool when it finished execution.

    The following screenshot shows an example configuration:

  2. Enter the following qTest Manager information:

    • qTest URL: The URL to access your qTest instance, such as https://demo.qtestnet.com.

    • API Key: This is the Integration with Jenkins token you obtained from the qTest Manager Resources page.

    • Select the Retrieve Data button to load data from qTest Manager.

    • Select the qTest project that the build is integrating with and where you want to submit test results.

    Now the configuration will look like the following:

Configure Options to Submit Test Results to Manager

  1. Configure the options to submit your test results to qTest Manager.

    • Submit test result to a Release as settings in your qTest Manager. By selecting this option, the qTest Plug-in will submit the test results to a Release in qTest Manager (refer to the Organizing Test Runs section above for more information on setting this up). Select a Release, which will be the target release of submitted Test Runs.

    • Submit Test Results to an Existing Container. By selecting this option, you allow the qTest Jenkins Plug-in to submit test results to a specific container in qTest Manager.

      • Container: Select a container where you want your test results to be submitted, such as a Release, a Test Cycle, or a Test Suite. The Plug-in will create a Test Suite directly under the selected container. Then it creates Test Runs underneath that Test Suite. It will then submit Test Logs to those Test Runs. If the Test Suite already exists, the Plug-in will not create duplicate Test Suites but will submit test results to that Test Suite.

      • Check this box to create a new test suite daily. If unchecked, a single test suite, with this job’s name, will be used: qTest Jenkins Plug-in will create a new Test Suite daily, then will create Test Runs underneath that test suite, and then will submit Test Logs to those Test Runs. If the Test Suite already exists (for example, if it was created by previous Jenkins built in the same day), then the Plug-in does not create a duplicate Test Suite but will submit test results to that Test Suite.

  2. (Optional) Environment: The selected Environment will be set as submitted Test Run's property.

    The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the newly selected Environment. The user must delete the existing Test Runs and create new ones if the user wants to inherit the Environment from the Test Suite.

  3. Overwrite existing test steps: Selected by default. When test results are submitted to qTest Manager, the Test Case steps are also updated in the log. If the original Test Case contained test step details with manual test steps, these details would get overwritten, since the test step logs would create newer Test Case versions.

    If you wish to retain test step details from the existing test case, you can deselect this option.

  4. Parse test results from testing tools: Selected by default. If selected, the Plug-in will scan for the XML files that contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the Plug-in will scan the whole project.

    This option is not available if you selected to execute Tosca tests.

  5. Utilize test results from the CI tool: In case there is one of the prior tasks that have already scanned and published test results to Jenkins, for example by using xUnit Plug-in, qTest Jenkins Plug-in will utilize the test results output from that task without scanning again.

    This option is not available if you selected to execute Tosca tests.

  6. Each JUnit Test Suite (class) = a qTest Test Case: Selected by default. A qTest Test Case is created from a JUnit Test Suite (class).

    This option is not available if you selected to execute Tosca tests.

  7. Each JUnit Test Case (method) = a qTest Test Case: A qTest Test Case is created from a JUnit Test Case (method).

    This option is not available if you selected to execute Tosca tests.

  8. Click the Generate Pipeline Script button to generate a pipeline script.

  9. The script will be generated in the text area below the Generate Pipeline Script button. Now you can copy the script and put it as a step in your pipeline scripts.

    Following is a sample Jenkins file that shows how the generated script is used in the entire pipeline scripts. The submitJUnitTestResultstoqTest step will submit the test results to qTest Manager once the step that executes the test is finished.

Bamboo Integration

Install the qTest Bamboo Plug-in

Install the qTest Plug-in for Bamboo from the Manage Add-ons page in Bamboo.

Configure qTest Bamboo Plug-in

  1. Navigate to qTest Manager and locate the Download qTest Resources Page  icon to open the page.

  2. Select the CI Tool Integration section to expand.

  3. Copy the API token from the Integration with Bamboo section. This allows Bamboo to access your qTest Manager instance.

    The API token is at a user level, which means it is different between qTest users.

  4. Navigate back to the Bamboo Plug-in Configuration page.

  5. Configure a build plan for your Bamboo server.

  6. Select a build stage under the plan.

  7. Select a build job under the stage.

  8. Add a new task to the selected build.

  9. On the pop-up, search for the qTest Integration task and add it to the stage.

    This task needs to be added to the job in which unit tests are executed.

  10. After the task is added, you will need to configure the task to connect to your qTest Manager instance.

  11. Enter the following qTest Manager information:

    • qTest URL: The URL to access your qTest instance, such as https://demo.qtestnet.com.

    • API Key: This is the Integration with Bamboo token you obtained from the qTest Manager Download Resources page.

  12. Select the Retrieve Projects button to load data from qTest Manager.

  13. Select the qTest project that the build is integrating with and where you want to submit test results.

  14. Select one Release that will be the target Release of submitted Test Runs.

  15. (Optional) The selected Environment will be set as submitted Test Run's property.

Notes

  • The selected Environment only affects the newly created Test Runs. It will not update the existing Test Runs with the new Environment if you have selected a new one. We are working on the fix for that and will release it soon.

  • The Plug-in will scan for the XML files that contain the test results. You can optionally specify a directory pattern where these XML files are located, using ANT Style Pattern. If not specified, the Plug-in will scan the whole project.

CI Integration Reports

CI Tool Integration Report

This is an export file that contains details of tests submitted from your CI server. It includes two views:

  • Build by Build: Details of all tests executed in each CI build

  • Test by Test: Execution results of each test through different CI builds

To download the export file:

  1. On the Test Execution tab, select the Export button.

  2. Select CI Tool Integration Report.

    • Select filters before exporting.

      The export is not filtered by what you select on Test Execution tree. It is only filtered by values selected in this pop-up.

Test Result Submission Report

On your CI tool side, we provide a report allowing you to track the status of test submissions to qTest after each build is completed.

If a failure occurs during test submission, the Failure Log will be included as a text file attachment in the Test Log Details.

  • Jenkins: Select a build project. Select the qTest Plug-in on the left panel.

  • Bamboo: Select a build plan. Open the qTest Plug-in tab.