Framework Evaluation Criteria
Support automation integration testing for UI (real/headless browser/device), API, performance in cross platforms.
- For the integration test, we don’t test the front-end or back-end individually. We need to verify the data with the integration between system, front-end & back-end. So the framework needs to be able to support all testing type and especially crossing browser, device & system.
Support CI integration with parallel execution.
- The testing framework needs to be able to integrate with our current CI/CD. Parallel execution will help reduce the build time, so we will be able to deploy quickly and have faster turnaround times for bugs and features.
Supports the concept of executable documentation for BDD (behavior-driven development)& DDT (data-driven testing) in modularization, maintainable and understandable test suites.
- BDD will help the test case & test suite easy to maintaining and understanding. It also helps communication between business and development is extremely focused as a result of common language.
- One of the most important concepts for effective test automation is modularization, it will help to create a sequence of test case only one and reuse as often as required in test script without rewriting the test all the time.
BDD Testing Framework Selection
| Gauge | Cucumber | |
|---|---|---|
| Language | Markdown | Gherkin |
| IDE & plugin support | Yes | Yes |
| Easy to integrate with CI/CD | Yes | Yes |
| Easy to use, quick to learn | No | No |
| Reusable, easy to maintain | Yes | Yes |
| Parallel execution | Built-in | 3rd party plugin |
| Customize reporting | Yes | Yes |
| Price | Open source & free | Open source & free |
Winner - Gauge:
- An open source lightweight cross-platform test automation tool with the ability to author test cases in the business language and have built-in parallel execution feature.
- Support BDD (Behavior-Driven Development) & CI(Continuous Integration) & report customization.
API Testing Framework Selection
| REST Assured | Postman | |
|---|---|---|
| Support BDD | Yes | 3rd third party |
| Support DDT | Yes | Limit |
| Easy to integrate with CI/CD | Yes | Yes |
| Easy to use, quick to learn | No | Yes |
| Reusable, easy to maintain | Yes | No |
| Customize reporting | Can be used with any customized/open source reporting tool. | No |
| Price | Open source & free | 8-21$ per user/month for professional collaboration & advanced features |
Winner - REST Assured:
- An open source Java-based Domain-Specific Language (DSL) that allows writing powerful, readable, and maintainable automated tests for RESTful APIs.
- Support testing and validating REST services in BDD (Behavior-Driven Development) / Gherkin format.
API Performance Testing Framework Selection
| Apache JMeter | Loadrunner | |
|---|---|---|
| Support DDT | Yes | Yes |
| Easy to integrate with CI/CD | Yes | Yes |
| Easy to use, quick to learn | Yes | No |
| Cross-platform | Yes | Windows, Linux |
| Reusable, easy to maintain | Yes | No |
| Customize reporting | Yes | Yes |
| Price | Open source & free | Free for first 50 virtual User |
Winner - Apache JMeter :
- Open source performance test runner & management framework may be used to test performance both on static and dynamic resources.
- Support load test functional behavior and measure performance. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.
Test Runner & Test Suite Management Framework Selection
| TestNG | JUnit | |
|---|---|---|
| Support DDT | Yes | Yes |
| Easy to integrate with CI/CD | Yes | Yes |
| Easy to use, quick to learn | No | No |
| Reusable, easy to maintain | Yes | Yes |
| Parallel execution | Yes | Yes |
| Price | Open source & free | Open source & free |
| Annotation support | Yes | Limit |
| Suite Test | Yes | Yes |
| Ignore Test | Yes | Yes |
| Exception Test | Yes | Yes |
| Timeout | Yes | Yes |
| Parameterized Test | Yes | Yes |
| Dependency Test | Yes | No |
| Support executing before & after all tests in the suite | Yes | No |
| Support executing before & after a test runs | Yes | No |
| Support executing before the first & last test method is invoked that belongs to any of these groups is invoked | Yes | No |
Winner - TestNG :
- Open source test runner framework which helps to run your tests in arbitrarily big thread pools with various policies available and flexible test configuration.
- Support DDT(Data-driven testing) & Test suite management.
Build Tool & Dependency ManagementFramework Selection
| Apache Maven | Grandle | |
|---|---|---|
| Easy to integrate with CI/CD | Yes | Yes |
| Easy to use, quick to learn | Yes | No |
| Build Script Language | XML | Groovy |
| Reusable, easy to maintain | Yes | Yes |
| Dependency Management | Yes | Yes |
| Dependency Scopes | Built-in | Custom |
| IDE & plugin support | Many | A little |
| Price | Open source & free | Open source & free |
Winner - Apache Maven
- The leading open source dependency management and build tool. It standardizes the software build process by articulating a project’s constitution.
- Software project management and comprehension tool in the concept of a project object model (POM), Maven can manage a project's build, reporting, and documentation from a central piece of information.
