Skip to main content

Test Intelligence overview

info

Test Intelligence applies to unit testing only. For other types of tests, use Run steps to run tests.

Testing is an important part of Continuous Integration (CI). Testing safeguards the quality of your product before shipping. However, test cycles often involve many tests, and it can take a significant amount of time for the tests to run. Additionally, the tests that run might be irrelevant to the code changes that triggered the build, and running all unit tests every time the code changes is expensive and time-consuming.

Harness Test Intelligence (TI) improves unit test time by running only the unit tests required to confirm the quality of the code changes that triggered the build. You can also use parallelism (test splitting) with TI to further optimize your test times.

How does Test Intelligence work?​

Test Intelligence uses test selection to run only those tests that are relevant to code changes. This includes changes to your software's code, as well as changes to your tests (new or modified tests). Instead of always running all unit tests, TI selects only the relevant subset of unit tests and skips the rest.

When you perform a pull request, TI uses the following metrics to select tests:

  • Changed code: TI queries Git to learn exactly which code changed in a specific build. TI uses this data to select tests that are associated directly or indirectly with the source code changes. TI selects these tests as part of the subset of the tests run in the pipeline. TI skips tests that aren't needed because there were no relevant code change.
    • Changes to certain files, such as build.gradle, pom.xml, and so on, can cause TI to select all tests due to the potential broad impact of these files. If a certain file is not relevant to the code you want to test, you can configure TI to ignore tests or files.
  • Changed tests: When a test is changed, TI selects and runs that test, even if the code the test covers hasn't changed.
  • New tests: When you add a new test, TI selects and runs that test. This ensures that the test is running successfully and also finds correlations between the new test and new/existing code.

TI is always up to date and syncs when you merge code to any branch.

After a build runs, TI gives you full visibility into which tests were selected and why. This can help you identify negative trends and gain insights to improve test quality and coverage. You can find the Test results and the TI call graph visualization on the Build details page. The call graph visualization shows the changed classes and methods that caused each test to be selected.

Test Intelligence architecture

Test Intelligence is comprised of a TI service, a Test Runner Agent, and the Run Tests step.

  • TI service: The TI service manages the data about repositories, git-commit graphs, test results, and call graphs. When a build runs, TI service uses a list of added/modified files with the call graph to identify which tests to run.
    • The TI service can receive real-time Git webhook notifications for any commit or merge. The TI service pulls the Git commit-graph and other metadata from Git for test selection.
    • When the TI Test Runner Agent sends a call graph generated from a PR, the TI service keeps that data in a staging area in case the PR doesn't get merged into the target branch (such as main). Once the TI receives the merge notification from Git, it updates and inserts the partial call graph with the target branch's call graph.
  • Test Runner Agent: The Test Runner Agent runs on the build infrastructure. It's responsible for communicating with the TI service. Whenever a Run Tests step initializes, the Test Runner Agent provides the TI service with the build number, commit-id, and other details, and the TI service returns the list of selected tests. The Test Runner Agent runs the selected tests. After all the tests run, the Agent parses the test results and uploads the results along with the newly-generated call graph.
  • Run Tests step: While you can also run tests in a Run step, to enable Test Intelligence, you must use the Run Tests step.
    • The Run Tests step is similar to the Run step, and it accepts additional test-specific information, such as the programming language of the source code being tested, build tools, and other parameters.
    • TI identifies the programming language and uses the Run Tests step to run the selected tests in that step's container. The Run Tests step, through the Test Runner Agent, parses the test results and returns the results to the TI service.

Supported codebases for Test Intelligence​

Test Intelligence is available for:

For other codebases, you can use Run steps to run tests.

Enable Test Intelligence​

Using TI doesn't require you to change your build and test processes. To enable TI, you must use a supported codebase and add a Run Tests step to your pipeline. For instructions and more information, go to:

You'll start seeing test selection and time savings on the second run after adding the Run Tests step. The first time you run a pipeline after adding the Run Tests step, Harness creates a baseline for test selection in future runs.

Once you start saving time with test selection, you can further optimize test times by enabling parallelism (test splitting) for TI. You can also configure TI to ignore tests or files.

Ignore tests or files​

If you want Test Intelligence to ignore certain tests or files, create a .ticonfig.yaml file in your codebase, and list the tests and files to ignore. For example:

config:
  ignore:
    - "README.md"
    - ".ticonfig.yaml"
    - "**/*.go"
    - "**/Dockerfile*"
    - "licenses/**/*"
    - "img/**/*"

View test reports and test selection​

Test results and test selection are reported on the Tests tab. The visualization call graph provides insights into why each test was selected. You can drill down into selected tests to examine selected classes/methods in more detail. For more information about test reports and test selection, go to View tests.

Troubleshoot Test Intelligence​

Go to the CI Knowledge Base for questions and issues related to Test Intelligence, including: