Framework Evaluation & Selection

Framework Evaluation Criteria

Support automation integration testing for UI (real/headless browser/device), API, performance in cross platforms.
- For the integration test, we don’t test the front-end or back-end individually. We need to verify the data with the integration between system, front-end & back-end. So the framework needs to be able to support all testing type and especially crossing browser, device & system.
Support CI integration with parallel execution.
- The testing framework needs to be able to integrate with our current CI/CD. Parallel execution will help reduce the build time, so we will be able to deploy quickly and have faster turnaround times for bugs and features.
Supports the concept of executable documentation for BDD (behavior-driven development)& DDT (data-driven testing) in modularization, maintainable and understandable test suites.
- BDD will help the test case & test suite easy to maintaining and understanding. It also helps communication between business and development is extremely focused as a result of common language.
- One of the most important concepts for effective test automation is modularization, it will help to create a sequence of test case only one and reuse as often as required in test script without rewriting the test all the time.

An open source lightweight cross-platform test automation tool with the ability to author test cases in the business language and have built-in parallel execution feature.
Support BDD (Behavior-Driven Development) & CI(Continuous Integration) & report customization.

	REST Assured	Postman
Support BDD	Yes	3rd third party
Support DDT	Yes	Limit
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	No	Yes
Reusable, easy to maintain	Yes	No
Customize reporting	Can be used with any customized/open source reporting tool.	No
Price	Open source & free	8-21$ per user/month for professional collaboration & advanced features

An open source Java-based Domain-Specific Language (DSL) that allows writing powerful, readable, and maintainable automated tests for RESTful APIs.
Support testing and validating REST services in BDD (Behavior-Driven Development) / Gherkin format.

Open source performance test runner & management framework may be used to test performance both on static and dynamic resources.
Support load test functional behavior and measure performance. It can be used to simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types.

	TestNG	JUnit
Support DDT	Yes	Yes
Easy to integrate with CI/CD	Yes	Yes
Easy to use, quick to learn	No	No
Reusable, easy to maintain	Yes	Yes
Parallel execution	Yes	Yes
Price	Open source & free	Open source & free
Annotation support	Yes	Limit
Suite Test	Yes	Yes
Ignore Test	Yes	Yes
Exception Test	Yes	Yes
Timeout	Yes	Yes
Parameterized Test	Yes	Yes
Dependency Test	Yes	No
Support executing before & after all tests in the suite	Yes	No
Support executing before & after a test runs	Yes	No
Support executing before the first & last test method is invoked that belongs to any of these groups is invoked	Yes	No

Open source test runner framework which helps to run your tests in arbitrarily big thread pools with various policies available and flexible test configuration.
Support DDT(Data-driven testing) & Test suite management.

The leading open source dependency management and build tool. It standardizes the software build process by articulating a project’s constitution.
Software project management and comprehension tool in the concept of a project object model (POM), Maven can manage a project's build, reporting, and documentation from a central piece of information.