Levels of Automation Excellence

 How effective is your automation test suite?

How impactful is it for your product and your team?
Do you know how to grow your test suite without sacrificing quality and performance?

These questions are surprisingly difficult to answer — especially when your entire suite feels like it’s constantly on fire, your tests are untrustworthy, and production bugs are popping up like they’re going out of style. (Just me?)

To bring some clarity — and because testers love pyramids — I created the Automation Maturity Pyramid as a way to measure automation impact.

First, let’s remember why we write automation tests in the first place. At the end-of-the-day, automation tests should support two simple missions:

• Increase product quality & confidence
• Accelerate development & deployment

So when we think about the pyramid and its phases, everything we do should ultimately align with those missions.

The pyramid has four levels of maturity:

1. Confidence — Trusting your test results.
2. Short-Term Impact — Creating value in daily development.
3. Speed of Development — Scaling automation without slowing down.
4. Long-Term Impact — Sustaining trust, visibility, and continuous improvement.

Each phase builds on the one below it. Later stages only unlock their benefits once the initial foundation is solid. The pyramid is both tool and type agnostic, meaning you can apply it to any automation suite, framework, or testing type that fits your needs.

Remember, this journey takes time. Think of the pyramid as a compass, not a checklist to rush through. If you’re starting fresh, it’ll guide you from the beginning. If you already have a suite, it’s a framework to measure current impact and decide what to tackle next.

Phase 1 — Confidence

A pyramid collapses without a strong base. The same is true with automation. If teams don’t trust the test failures (or even successes), everything else becomes meaningless.

When results are unreliable, people stop acting on them. And when tests are ignored, automation loses its purpose. In many ways, unreliable automation is often worse than not having any at all.

The Tests Must Pass

Failures will happen. That’s not the issue. The danger is when teams normalize broken tests or flaky failures. Every red test should be taken seriously: investigated, understood, and resolved. While there are exceptions, the default culture must be: stop and fix. Adopt the mindset “all tests must pass”, and technical debt will quickly diminish before it starts. A mature automation test suite starts with an accountable mindset.

What Undermines Confidence

• Flakiness: Tests that pass or fail inconsistently without code changes. Common causes include race-condition, non-deterministic app behavior, dependent tests or poor test data management.
• Environment Instability: Where you will run your tests matter, especially if multiple options are needed. Can you guarantee tests will run reliably across all environments?
• Weak Data Strategies: Do tests always have the data they need? Is it static or dynamic? A strong data strategy reduces countless downstream failures. My favorite data management is through programmatic control.

Phase 1 is about establishing trust. Once failures are credible and environments stable, your suite stops being noise and starts being a safety net. A small, confident test suite is more impactful than a large, unstable one. Some actions items to consider:

• Research and implement flake-reduction practices for your tool of choice
• Create a culture of accountability: quarantine flaky tests and resolve them quickly
• Write tests environment-agnostically
• Define a consistent test data strategy that works across environments

If you’ve done these, you’re ready for Phase 2.

Phase 2 — Short-Term Impact

With trust established, the next step is to make automation useful right now. Tests should provide fast feedback and reduce risk during daily development.

If tests only run occasionally or if results arrive too late to act on, they don’t influence decision-making. The goal is to make automation an indispensable partner for developers, not a background chore.

This phase is all about defining an initial CI/CD strategy that suites your team’s development processes.

CI/CD Strategy

A good rule: the closer tests run to code changes, the more valuable they are. Running suites pre-merge ensures failures tie directly to specific commits, not multiple layers of changes. Fewer variables mean quicker triage.

Nightly or scheduled runs still have a place — especially for full regressions, but the longer the gap between code and results, the harder it is to debug.

Some common strategies:

• Pre-merge Tests: Run in under ~10 minutes. Cover critical paths first, then expand with performance in mind.
• Full Nightly Regression: Capture broader coverage where speed isn’t urgent.
• Custom Tag-Based Gates: Sub-groups of tests run based on criteria.

Results Visibility

Running tests is meaningless if no one notices the outcomes. Ensure results are clear, fast, and shared.

Every suite should generate artifacts accessible to all engineers. This includes screenshots, video, error logs and any other additional test information. Without proper artifacts, debugging failures becomes exponentially harder. Additionally, notifications should be immediate and integrated into tools your teams already use.

A professional rule of mine— act like Veruca Salt from Willy Wonka:
“I want those results and I want them now!”

Remember, Phase 2 is about usefulness. Once tests deliver fast, actionable feedback, they directly help teams ship better code, quicker. Developers know within minutes when a real-bug is introduced. Testers know when flake is first introduced, for immediate remediation.

Stick to the mantra: “all tests must pass”.

Once you start getting short-term feedback from your tests, it’s time to optimize them.

Phase 3 — Speed of Development

Once automation is trusted and embedded in the workflow, the focus shifts to efficiency. The question becomes: how can automation help us move faster without cutting corners?

At small scale, almost any automation adds value. But as suites grow, inefficiency turns automation into a bottleneck. Tests that take hours to run or are painful to debug become blockers instead of enablers. This phase has three areas of focus: writing, debugging and executing tests.

Write Tests Faster

Writing tests faster primarily comes down to test organization and structure. Expanding further:

• Standardize Structure: Use any pattern that makes sense to you and don’t worry about perfection. Any organization beats spaghetti-code chaos. Optimize over-time.
• Reuse Aggressively: Create helpers, builders, and shared libraries for scaleability.
• Proactive Test Planning: Review product tickets early to avoid last-minute gaps.
• Use AI-assisted Tooling: Just do it. There’s no excuse not to use AI anymore. Embrace our new overlords!
• Document: Look, we all know it sucks…but providing guides and common gotchas reduce ramp-up time as the team grows. What would past you wish they had when they first onboarded?

Debug Tests Faster

Test failures will happen so response time makes or breaks a suite’s value.

• Prioritize Readability: Choose clarity over cleverness; smaller, focused tests are easier to diagnose. Always write tests with future you in mind. “Will this make sense to me in six months?”.
• Reduce Variables: Run tests as close to the change as possible (prioritize pre-merge if not already implemented).
• Culture of Accountability: Build a habit of immediate triage: treat all fails with the same urgency so at least some resolution occurs.
• Improved Artifact Tools: Interactive runners, browser devtools, and in-depth logs are gold. Improve artifacts as needed.

Run Tests Faster

This one is simple. How fast do our tests run? Repeat after me: “Nobody brags about a three-hour test suite”. As the test suite grows, will the team still get quick value without slowing down the process?

• Parallelize: Split suites across multiple machines or containers. A must for pre-merge pipelines.
• Subset Tests: Run critical paths first; save broader regressions for later. Customize based on need and overall test performance.
• Optimize Code: Remove hard-coded waits, reduce unnecessary DOM interactions, apply tool best practices.

Phase 3 is about efficiency. Automation should accelerate delivery, not drag it down. When done well, it enables rapid iteration and frequent, confident releases. All of a sudden our monthly releases can now be reduced to weekly. Then daily. Then maybe even multiple times a day, if you’re feeling extra daring. All thanks to your automation test suite.

You deserve a raise.

Phase 4 — Long-Term Impact

The final phase is about sustainability. Once automation is fast, useful, and trusted, it must also deliver long-term value.

Teams and products evolve. Without continuous investment, automation rots: tests get flaky, results get ignored, and the pyramid crumbles. Which is all super sad. Professional advice, don’t be sad.

Long-term impact ensures automation remains a source of truth while showcasing just how cool your team is.

Metrics Inform, Not Punish

This phase is purely about responding to metrics, but use them wisely. Metrics should guide investment, not assign blame. Focus on impactful metrics that guide your automation roadmap. Simply, you don’t know what to improve if you don’t know what’s ineffective.

Some Suggestions:

• Test Coverage: Directional, not definitive. Pair with quality checks.
• Pass/fail and flake rates: Indicators of credibility.
• Execution time: Is the suite scaling with the team?
• Time-to-resolution (TTR): How quickly do teams fix failures?
• Defect detection efficiency (DDE): Percentage of bugs caught by automation.

If possible, consider augmenting these with a dashboard where visibility is further increased. Visual trends make it easier to consume historical trends and identify weaknesses. Plus bar graphs are fun and line graphs always look convincing. Don’t even threaten me with a good time and bring up pie charts.

This phase is small but important. It’s the culmination of all the previous phases, and purely intended to bring visibility into how well things went in the previous phases. It drives future revisions and ensures the test suite is never stagnant in it’s impact.

Phase 4 is all about trust at scale. Mature automation creates transparency, informs investment, and continues to improve over time.

Putting It All Together

The Automation Maturity Pyramid is a lot smaller than the Pyramids of Giza but much more relatable since those are real and in Egypt and this is thought-leadership and about testing. Just to clarify any confusion to this point.

But seriously, it’s about measuring your impact, one phase at a time. Building a successful automation test suite is hard without proper guidance. There’s many technical steps and failures can quickly become overwhelming and frustrating.

To recap:

• Confidence First: You have to trust your tests, always. The rest will follow.
• Early Wins: No matter the test suite size, obtain value. Start catching real issues.
• Take small steps: Steady improvements compound into big gains. Efficiency is a learning curve and only obtained through experience.
• Welcome Failures: Hello failures, come on it. Have a seat. Let’s talk about how you’re making my current life bad so we can make my future life good.
• Celebrate Progress: Building a reliable, impactful suite is a team achievement. Be proud of that green test run, those first 100 tests, or the first real-bug your suite caught. You’re a rockstar, genuinely.

Done well, automation isn’t overhead — it’s a strategic advantage. Build a base of trust, create fast feedback loops, optimize for speed, and commit to long-term transparency. That’s how you turn test automation into a driver of product success.

Best of luck in your climb. And as always, happy testing.

Handling Large Payloads in RestAssured: Best Practices and Examples

Introduction

When testing APIs with RestAssured, it's common to encounter scenarios that require sending large JSON or XML payloads. This is particularly relevant for bulk data uploads, complex configurations, or nested objects. If not handled effectively, large payloads can lead to code clutter, memory inefficiency, and maintenance challenges.

This blog outlines the common challenges and provides best practices to manage large payloads efficiently in RestAssured.

Challenges with Large Payloads

• Code Readability: Hardcoding large payloads directly in test methods makes code messy and difficult to maintain.

• Maintainability: Any change in the payload requires updates to the test code and possible redeployments.

• Performance: Large payloads can increase memory usage and slow down test execution if not optimized.

• Validation Complexity: Verifying large responses requires structured and scalable approaches.

Best Practices to Handle Large Payloads in RestAssured

1. Externalize Payloads in Files
Store payloads in separate files (e.g., .json or .xml) and load them at runtime.

• Advantages: Cleaner code, easy updates, and version control.

Example:

import io.restassured.RestAssured;

import java.nio.file.Files;

import java.nio.file.Paths;

public class LargePayloadTest {

    public static void main(String[] args) throws Exception {

        String jsonBody = new String(Files.readAllBytes(Paths.get("src/test/resources/largePayload.json")));

        RestAssured.given()

            .header("Content-Type", "application/json")

            .body(jsonBody)

            .when()

            .post("https://api.example.com/upload")

            .then()

            .statusCode(200);

    }

}

2. Use POJOs with Serialization
Represent payloads as Java objects and let RestAssured serialize them using Jackson or Gson.

• Advantages: Strong typing, compile-time checks, and easy field modifications.

Example:

class Employee {

    public String name;

    public int age;

    public List<String> skills;

}

Employee emp = new Employee();

emp.name = "John";

emp.age = 35;

emp.skills = Arrays.asList("Java", "Selenium", "RestAssured");

RestAssured.given()

    .contentType("application/json")

    .body(emp)

    .post("/employees")

    .then()

    .statusCode(201);

3. Use Template Engines for Dynamic Payloads
When most of the payload remains static but some fields change, template engines or simple string replacements work well.

• Tools: Apache Velocity, FreeMarker, or String.format().

Example:

String template = new String(Files.readAllBytes(Paths.get("template.json")));

String payload = template.replace("${username}", "john.doe")

                         .replace("${email}", "john@example.com");

4. Compress Large Payloads (If Supported)
If your API supports compression, use GZIP to reduce payload size and network latency.

Example:

RestAssured.given()

    .contentType("application/json")

    .header("Content-Encoding", "gzip")

    .body(CompressedUtils.gzip(jsonBody))

    .post("/bulkUpload");

5. Streaming Large Files
Avoid loading entire files into memory by streaming them directly during uploads.

Example:

File largeFile = new File("largeData.json");

RestAssured.given()

    .multiPart("file", largeFile)

    .post("/upload")

    .then()

    .statusCode(200);

When to Choose Which Approach

• Use external files for static or semi-static payloads.

• Use POJOs for strongly typed, programmatically generated data.

• Use templates for partially dynamic payloads.

• Use compression or streaming for very large payloads.

Summary
To handle large payloads in RestAssured efficiently:

• Avoid hardcoding payloads.

• Externalize or serialize data for cleaner, maintainable code.

• Use templates for flexibility and compression or streaming for very large files.

• Choose the right approach based on payload type and test goals.

🚀 Introducing the Universal API Testing Tool — Built to Catch What Manual Testing Misses

In today’s software-driven world, APIs are everywhere — powering everything from mobile apps to microservices. But with complexity comes risk. A single missed edge case in an API can crash systems, leak data, or block users. That’s a huge problem.

After years of working on high-scale automation and quality engineering projects, I decided to build something that tackles this challenge head-on:

👉 A Universal API Testing Tool powered by automation, combinatorial logic, and schema intelligence.

This tool is designed not just for test engineers — but for anyone who wants to bulletproof their APIs and catch critical bugs before they reach production.

---

🔍 The Problem with Manual API Testing

Let’s face it: manual API testing, or even scripted testing with fixed payloads, leaves massive blind spots. Here’s what I’ve consistently seen across projects:

• 🔁 Happy path bias: Most tests cover only expected (ideal) scenarios.

• ❌ Boundary and edge cases are rarely tested thoroughly.

• 🧱 Schema mismatches account for over 60% of integration failures.

• 🔄 Complex, nested JSON responses break traditional test logic.

Even with the best intentions, manual testing only touches ~15% of real-world possibilities. The rest? They’re left to chance — and chance has a high failure rate in production.

---

💡 Enter: The Universal API Testing Tool

This tool was created to turn a single API request + sample response into a powerful battery of intelligent, automated test cases. And it does this without relying on manually authored test scripts.

Let’s break down its four core pillars:

---

🔁 1. Auto-Schema Derivation

Goal: Ensure every response conforms to an expected structure — even when you didn’t write the schema.

• Parses sample responses and infers schema rules dynamically

• Detects type mismatches, missing fields, and violations of constraints

• Supports deeply nested objects, arrays, and edge data structures

• Validates responses against actual usage, not just formal docs

🔧 Think of it like “JSON Schema meets runtime intelligence.”

---

🧪 2. Combinatorial Test Generation

Goal: Generate hundreds of valid and invalid test cases automatically from a single endpoint.

• Creates diverse combinations of optional/required fields

• Performs boundary testing using real-world data types

• Generates edge case payloads with minimal human input

• Helps you shift-left testing without writing 100 test cases by hand

📈 This is where real coverage is achieved — not through effort, but through automation.

---

📜 3. Real-Time JSON Logging

Goal: Provide debuggable, structured insights into each request/response pair.

• Captures and logs full payloads with status codes, headers, and durations

• Classifies errors by type: schema, performance, auth, timeout, etc.

• Fully CI/CD compatible — ready for pipeline integration

🧩 Imagine instantly knowing which combination failed, why it failed, and what payload triggered it.

---

🔐 4. Advanced Security Testing

Goal: Scan APIs for common and high-risk vulnerabilities without writing separate security scripts.

• Built-in detection for:

  • XSS, SQL Injection, Command Injection

  • Path Traversal, Authentication Bypass

  • Regex-based scans for sensitive patterns (UUIDs, tokens, emails)

• Flags anomalies early during development or staging

🛡️ You don’t need a separate security audit to find the obvious vulnerabilities anymore.

---

⚙️ How It Works (Under the Hood)

• Developed in Python, using robust schema libraries and custom validation logic

• Accepts a simple cURL command or Postman export as input

• Automatically generates:

  • Schema validators

  • Test payloads

  • Execution reports

• Debug mode shows complete request/response cycles for every test case

---

📈 What You Can Expect

The tool is in developer preview stage — meaning results will vary based on use case — but here’s what early adopters and dev teams can expect:

• ⏱️ Save 70–80% of manual testing time

• 🐞 Catch 2–3x more bugs by testing combinations humans often miss

• ⚡ Reduce integration testing time from days to hours

• 🔒 Get built-in security scans with every API run — no extra work required

---

🧰 Try It Yourself

🔗 GitHub Repository

👉 github.com/nsharmapunjab/frameworks_and_tools/tree/main/apitester

---

💬 Your Turn: What’s Your Biggest API Testing Challenge?

I’m actively working on v2 of this tool — with plugin support, OpenAPI integration, and enhanced reporting. But I want to build what developers and testers actually need.

So tell me:

➡️ What’s the most frustrating part of API testing in your projects?

Drop a comment or DM me. I’d love to learn from your use cases.

---

👋 Work With Me

Need help building test automation frameworks, prepping for QA interviews, or implementing CI/CD quality gates?

📞 Book a 1:1 consultation: 👉 topmate.io/nitin_sharma53

---

Thanks for reading — and if you found this useful, share it with your dev or QA team. Let’s raise the bar for API quality, together.

#APITesting #AutomationEngineering #QualityAssurance #DevOps #OpenSource #TestAutomation #PythonTools #API #SDET #NitinSharmaTools

🕵️‍♂️ SVG vs Shadow DOM in Selenium: A Tester’s Guide with Real-World Examples

Have you ever clicked an element in Selenium, only to watch nothing happen—again and again? Welcome to the world of SVGs and Shadow DOMs, where traditional locators fail and frustration often begins.

In this post, we’ll demystify these tricky elements, explain how to work with them in Selenium (Java), and walk through real-world examples that every automation engineer should know.

---

🧩 What Are SVG and Shadow DOM?

Appium Architecture - Core Concepts

What is Appium?

It’s a NodeJS based open-source tool for automating mobile applications. It supports native, mobile web, and hybrid applications on iOS mobile, Android mobile, and Windows desktop platforms.

Using Appium, you can run automated tests on physical devices or emulators, or both.

Let’s understand the above Appium architecture diagram.

• Appium is a client-server architecture. The Appium server communicates with the client through the HTTP JSONWire Protocol using JSON objects.
• Once it receives the request, it creates a session and returns the session ID, which will be used for communication so that all automation actions will be performed in the context of the created session.
• Appium uses the UIAutomator test framework to execute commands on Android devices and emulators.
• Appium uses the XCUITest test framework to execute commands on Apple mobile devices and simulators.
• Appium uses WinAppDriver to execute commands for Windows Desktop apps. It is bundled with Appium and does not need to be installed separately.

Appium - Android visual interaction flow

Let’s understand the interaction flow between the code and the Android device via the Appium server.

• The client sends the request to the Appium server through the HTTP JSONWire Protocol using JSON objects.
• Appium sends the request to UIAutomator2.
• UIAutomator2 communicates to a real device/simulator using bootstrap.jar which acts as a TCP server.
• bootstrap.jar executes the command on the device and sends the response back.
• Appium server sends back the command execution response to the client.

Appium - iOS visual interaction flow

Let’s understand the interaction flow between the code and the iOS device via the Appium server.

• The client sends the request to the Appium server through the HTTP JSONWire Protocol using JSON objects.
• Appium sends the request to XCUITest.
• XCUITest communicates to a real device/simulator using bootstrap.js which acts as a TCP server.
• bootstrap.js executes the command on the device and sends the response back.
• The Appium server sends the command execution response to the client.

Whiteboard Sessions

• IOS flow architecture

• Android flow architecture

• Drivers which appium supports
• UI Automator2 (Android)
• Espresso (Android)
• WinApp (Windows)
• MAC Driver (Mac OS)
• XCUITest (IOS above 9.3 version)
• UI Automation (IOS below 9.3 version)
• Tizen (for samsung)

Happy Learning :)