Jenkins as the Test Reporting Framework

Very few and matured companies are able to invest on the development of the Test Reporting Frameworks. At Komli we realised the benefits & value it offers.Instead of buying some commercial tools we went with the approach of extending Jenkins for the same. This talk is about our experience regarding the same.
Test Reporting Framework are the ultimate utilities reached by very few companies in the world likes of Google & Facebook. It require significant investment in terms money and resources. To optimize on both we tried the approach of extending Jenkins for the same purpose.
We like to share our experiences with respect to this journey. We will talk about different challenges we faced in doing so and how we solved them.
We strongly believe that other startup can use our experience and get benefit from this immensely.

WHY TEST REPORTING FRAMEWORK?

1. To help with informed decision, whether to go ahead with production deployment of the build
2. To analyze whether sufficient testing being done

• What are code changes?
• What are the corresponding automation changes w.r.t test-cases added/deleted/modified
• What is the test automation run trend w.r.t trend on Release basis

REPORTING FRAMEWORK REQUIREMENTS

• Web based Tool
• Cost Effective
• Easy to maintain
• Integration with Test Case Execution Frameworks

SOME OF PAID TEST REPORTING FRAMEWORKS AVAILABLE

• TestRail
• Xqual
• Test Collab

SOLUTION TO THE ABOVE PROBLEMS :- JENKINS

• Continuous Integration server that allows to run multiple automation suites at the same time.
• Supports SVN, Git and about any other SCM tools you can think of via plugins

WHY JENKINS

• Web based
• No additional cost, we are already dependent on it as Test-Case Execution Framework
• Well supported
• Highly customizable

JENKINS ON THE RELEASE BASIS

COMMON JENKINS TERMS

• Project
• Builds

NEW TERMS INTRODUCED

• Benchmark Project
• Benchmark Builds 

BENCHMARK PROJECT

• The project in Jenkins where we maintain the automation results on the basis of releases done to production.

BENCHMARK BUILD

• Build which has best results
• The one whose final status we want to maintain in the benchmark project

WHAT IS ALREADY AVAILABLE IN JENKINS

• Show automation run trend
• Integration with SCM tool show code diffs

CHALLENGES

1. Integration : Want everything in the single screen (single project in Jenkins)
2. Benchmark Build : We don't know priori that given build is the best build to run

• Don't want to waste resources and time to execute automation run again once we have decided on the build to deploy on production.
• Intermittent Failures : Want to maintain their status as Passed for a given release
• If we have taken the approach of copying results from one project to other project (benchmark project).

OUR APPROACH

• Git Plugin for Code Diff
• Custom Java Script for Test-Cases diff
• Custom Java Script for overriding intermittent failure status
• CopyArtifact Plugin to migrate results
• Custom Groovy Script for re-generating the TestNG result trends

INTERMITTENT FAILURES

 

PARAMETERS TO COPY BENCHMARK BUILD

Slideshare Link for Presentation at RootConf