The AI Test Reporter is a powerful tool that works with any test framework and leverages your existing accounts with AI language models like OpenAI GPT, Azure and Anthropic Claude to create concise, human-readable summaries of test results, highlighting likely causes and suggesting possible solutions to help your team resolve issues faster.

TL:DR — Check out the AI Test Reporter to generate an AI test report with any testing framework!

Can You Help Us Grow? 🙏

Before diving into how this works, we really need your help to grow! Please consider giving our GitHub organization a follow and support our mission to enhance test reporting with AI. 🙌

Key Features

Immediate Insights: Print summaries directly to the stdout after running your tests.
Seamless Integration: Integrate with developer tools like GitHub, Slack and more for effortless collaboration.
Universal Compatibility: Works with all major testing frameworks via standardized CTRF reports.
AI Summary JSON Report: Add an AI summary to your CTRF JSON report, enabling further processing and integration with other tools and workflows.

Here is an example of the stdout:

In this blog post, we’ll walk you through how to set up and use the AI Test Reporter

OK, Let’s Go!!

To begin using the AI Test Reporter, you’ll need a funded API account with OpenAI, Anthropic, or another leading AI provider that offers API access. Ensure your account allows you to generate an API access token, as this token is essential for integrating the AI capabilities into your testing workflow.

Step 1: Install the CTRF Reporter

First, you need to generate a CTRF test report. CTRF reporters are available for most modern test frameworks, and you can also convert a JUnit report.

Step 2: Generate an AI Summary

Now that you have your CTRF test report, you can use the AI Test Reporter to generate AI-powered summaries for any failed tests. While the tool supports multiple AI providers, we’ll use OpenAI in this example.

Execute the following command in your terminal:

npx ai-ctrf openai <path-to-ctrf-report>

Replace with the path to your CTRF report file, typically ctrf-report.json.

Since the package interacts with the OpenAI API, you will need to set the OPENAI_API_KEY environment variable. For further details on configuration options, check out the AI Test Reporter documentation.

After running the command, the AI Test Reporter will analyze your CTRF report and summarise any failed tests. The summary will be printed directly to your console, highlighting likely causes and suggesting possible solutions to help you debug more efficiently.

Seamless Integration with CI/CD Tools

One of the standout features of AI Test Reporter is that the AI-generated summary becomes a part of your CTRF JSON report. This opens up limitless opportunities for integrating the report into various stages of your CI/CD pipelines and developer workflows. With a standardized data exchange format, you have access to all the relevant test information, including the AI summary, which you can easily share across your toolchain. Whether it’s posting the summary to pull requests, tickets, or communication platforms, the possibilities are vast.

CTRF comes with built-in support for many platforms. In this post, I’ll focus on two: GitHub and Slack.

We’d love to hear how you’re using your CTRF JSON report in your workflows — share your experience in the comments!

Publish AI Test Summary to GitHub Actions

You can display the AI-generated test summary in your GitHub Actions workflow by using the GitHub Test Reporter. Add the following to your workflow YAML file:

- name: Publish CTRF AI Test Summary Results
  run: npx github-actions-ctrf ai <path-to-ctrf-report>
  if: always()

You will have an AI-generated failed test summary, seamlessly integrated into your GitHub Actions workflow summary

If you’d like to add a comment with your AI summary to an associated pull request, simply use the --pr argument:

- name: Publish CTRF AI Test Summary Results
  run: npx github-actions-ctrf ai <path-to-ctrf-report> --pr
  if: always()

You’ll see an AI summary as a comment on the related pull request — ensuring the insights are visible where they’re most needed.

Send AI test summary to your Slack Channel

Use the Slack Test Reporter to send AI generated test summaries directly to your Slack channel. This integration allows team members to see failed test details quickly and suggested solutions without leaving their communication platform.

npx slack-ctrf ai <path-to-ctrf-report>

More CI/CD integrations are being built, checkout the AI Test Reporter documentation for all integrations

How Does This Work?

CTRF (Common Test Report Format) is a standardized test report schema that serves as a bridge between various testing frameworks and developer tools. It includes a collection of tooling to integrate with platforms like GitHub Actions and more. Among its various properties, the CTRF schema supports an ai property for AI-generated summaries.

Here’s an example of what a CTRF report test object looks like with the AI property

{
  "name": "should display title",
  "status": "failed",
  "duration": 1200,
  "message": "error message",
  "trace": "error trace",
  "ai": "AI generated test summary"
}

In this schema:

name: The name of the test.
status: The result of the test (e.g., failed).
duration: The time the test took to run (in milliseconds).
message: The error message from the failed test.
trace: The stack trace of the error.
ai: The AI-generated test summary, provides a concise explanation of the likely cause of failure and possible solutions.

By using CTRF, you can easily integrate AI-generated test summaries into your workflow. The ai property is filled with the summary provided by AI models such as OpenAI or Anthropic, which can then be displayed in your GitHub Actions summary or added as a comment to a pull request.

You can do so much with CTRF, and the tooling is constantly evolving to better support your testing and development workflows.

If you found this useful, remember to follow the organization and give a star to the repositories you find useful.

🧡 Your support helps the library grow and is truly appreciated!