Test Intelligence overview
Test Intelligence applies to unit testing only. For other types of tests, use Run steps to run tests.
Testing is an important part of Continuous Integration (CI). Testing safeguards the quality of your product before shipping. However, test cycles often involve many tests, and it can take a significant amount of time for the tests to run. Additionally, the tests that run might be irrelevant to the code changes that triggered the build, and running all unit tests every time the code changes is expensive and time-consuming.
Harness Test Intelligence (TI) improves unit test time by running only the unit tests required to confirm the quality of the code changes that triggered the build. You can also use parallelism (test splitting) with TI to further optimize your test times.
How does Test Intelligence work?
Test Intelligence uses test selection to run only those tests that are relevant to code changes. This includes changes to your software's code, as well as changes to your tests (new or modified tests). Instead of always running all unit tests, TI selects only the relevant subset of unit tests and skips the rest.
When you perform a pull request, TI uses the following metrics to select tests:
- Changed code: TI queries Git to learn exactly which code changed in a specific build. TI uses this data to select tests that are associated directly or indirectly with the source code changes. TI selects these tests as part of the subset of the tests run in the pipeline. TI skips tests that aren't needed because there were no relevant code change.
- Changes to certain files, such as
build.gradle
,pom.xml
, and so on, can cause TI to select all tests due to the potential broad impact of these files. If a certain file is not relevant to the code you want to test, you can configure TI to ignore tests or files.
- Changes to certain files, such as
- Changed tests: When a test is changed, TI selects and runs that test, even if the code the test covers hasn't changed.
- New tests: When you add a new test, TI selects and runs that test. This ensures that the test is running successfully and also finds correlations between the new test and new/existing code.
TI is always up to date and syncs when you merge code to any branch.
After a build runs, TI gives you full visibility into which tests were selected and why. This can help you identify negative trends and gain insights to improve test quality and coverage. You can find the Test results and the TI call graph visualization on the Build details page. The call graph visualization shows the changed classes and methods that caused each test to be selected.
Test Intelligence architecture
Supported codebases for Test Intelligence
Test Intelligence is available for:
For other codebases, you can use Run steps to run tests.
Enable Test Intelligence
Using TI doesn't require you to change your build and test processes. To enable TI, you must use a supported codebase and add a Run Tests step to your pipeline. For instructions and more information, go to:
You'll start seeing test selection and time savings on the second run after adding the Run Tests step. The first time you run a pipeline after adding the Run Tests step, Harness creates a baseline for test selection in future runs.
Once you start saving time with test selection, you can further optimize test times by enabling parallelism (test splitting) for TI. You can also configure TI to ignore tests or files.
Ignore tests or files
If you want Test Intelligence to ignore certain tests or files, create a .ticonfig.yaml
file in your codebase, and list the tests and files to ignore. For example:
config:
ignore:
- "README.md"
- ".ticonfig.yaml"
- "**/*.go"
- "**/Dockerfile*"
- "licenses/**/*"
- "img/**/*"
View test reports and test selection
Test results and test selection are reported on the Tests tab. The visualization call graph provides insights into why each test was selected. You can drill down into selected tests to examine selected classes/methods in more detail. For more information about test reports and test selection, go to View tests.
Troubleshoot Test Intelligence
Go to the CI Knowledge Base for questions and issues related to Test Intelligence, including:
- Does Test Intelligence split tests? Can I use parallelism with Test Intelligence?
- Test Intelligence call graph is empty.
- If the Run Tests step fails, does the Post-Command script run?
- Ruby Test Intelligence can't find rspec helper file.
- Test Intelligence fails due to Bazel not installed, but the container image has Bazel.
- Does Test Intelligence support dynamic code?
- Errors when running TI on Python code.
- Errors when running TI with Maven.
- Gradle version not compatible with Test Intelligence.