Back to Portfolio
Agentic AISoftware TestingQAEnterprise SoftwareAI Architecture

Scripts, Services, and Agents: What Six Months in QA Pre-Sales Taught Me About Unbundling Testing

A pre-sales engineer's field notes on how agentic AI is splitting enterprise QA into managed services and embeddable tools — and what the research says about where full autonomy still breaks.

Bijesh Singha
Bijesh Singha
·12 min read

In the final stretch of my MBA, I joined Webomates as a part-time Product and Solutions Engineer on the technical pre-sales side, starting in January 2026. When I graduated in April, I came on full-time and stayed through July. My job was to sit across from engineering leaders — VPs of Engineering, Heads of QA, DevOps leads, mostly at US tech companies — and walk them through what AI-driven testing could actually do for their pipelines. I ran a lot of demos. I sat in on a lot of discovery calls. And underneath almost every one of those conversations was the same unresolved tension: does this team want to hand off testing entirely, or do they want a tool they can own?

In the weeks since wrapping up there, I've been buried in RAG pipelines and long-context benchmarks for my own portfolio work, chasing a version of that same question in a different domain: when does it make sense to buy a fully managed system, and when does it make more sense to embed a narrow, autonomous tool into something you already control? I wrote about that tension for retrieval architectures a while back. Coming back to it now, I realize I'd already watched this exact debate play out in real time, on calls, months before I had language for it.

Software QA in 2026 is going through its own version of the "unbundling" moment that's already reshaped content delivery and customer support. For two decades, the market for outsourced testing was dominated by full-service providers: hand over your application, get a tested build back. Agentic AI is pulling that model apart, and having sat on the sales-engineering side of that shift, I think the reasons why say something bigger about how AI-native markets settle into shape.

The maintenance tax that broke scripted testing

Traditional automation — Selenium, Playwright scripts, record-and-playback tools — is path-dependent. A test says "click the button with this exact locator," and if a developer renames a CSS class or moves an element six pixels to the left, the test breaks, whether or not anything is actually wrong with the product. Teams running mature suites reportedly lose something like 40–60% of their QA engineering time to exactly this kind of upkeep — fixing tests broken by routine UI changes rather than catching real bugs.

I heard this cost described in almost identical terms on call after call, in different words each time: a QA lead would tell me their team spent more hours babysitting a suite of thousands of Selenium scripts than they spent actually investigating real defects. Every sprint, someone renamed a class or restructured a component, and every sprint, a chunk of the regression suite turned red for reasons that had nothing to do with product quality. That's not an edge case — it's close to the default experience of running automation at scale, which is exactly why "self-healing" became the single most-requested feature in nearly every demo I gave.

That brittleness is why the industry has moved through fairly distinct generations. Manual testing gave way to record-and-playback tools, which gave way to scripted frameworks like Selenium (roughly 2004–2020). Low-code platforms such as Mabl and Testim sped up authoring through 2020–2024, but the underlying coupling to the DOM never went away. Forrester eventually renamed its entire market category from "Continuous Automation Testing Platforms" to "Autonomous Testing Platforms," after concluding that scripted approaches had plateaued at around a quarter of test coverage industry-wide — you simply can't script your way past that ceiling when releases ship daily.

What's usually marketed as a single "self-healing" feature is actually a fallback chain, and the ordering matters. A locator fails, and the system first tries other structural attributes on the same element — ID, ARIA role, position in the DOM tree. If that fails, it falls back to semantic or visual matching — does something that looks and behaves like a "submit button" exist nearby, even if its class name changed. Only when both of those come up empty does a well-designed system reach for the expensive option: LLM- or embedding-based matching against a screenshot or accessibility tree. That ordering is the whole trick — it's what keeps a self-healing suite fast and affordable at regression scale instead of routing every element lookup through a model call.

The Three Eras of Test Automation

What "agentic" actually means (and what it doesn't)

The word "agentic" gets stretched to cover almost anything with an AI feature attached to it, so it's worth being precise. A useful three-tier way to think about it:

  • AI-assisted — the AI helps a human write scripts faster: autocomplete, code suggestions.
  • AI-augmented — the AI handles one specific task autonomously, like element detection or locator repair, but a human still directs the overall test.
  • Agentic — the AI operates on a goal, not a script, and owns the full loop: understanding requirements, generating tests, executing them, diagnosing failures, and adapting — without waiting for instruction at every step.

That third tier runs on a continuous reasoning loop, not unlike the plan-and-act loops you see in coding agents: perceive the current state of the application, reason about what to do next, act on the interface, and verify whether the outcome matches intent. When it doesn't match, a well-designed agent doesn't just fail loudly — it re-plans and tries again.

In practice, most agentic testing platforms aren't a single agent running that loop end to end — they're a small crew of specialized agents, in roughly the same orchestrator-worker pattern popularized by frameworks like CrewAI and AutoGen. One agent reads a user story or spec and generates test cases. A second executes them against the live application. A third clusters failures and reasons about whether a given failure is a real regression, a flaky environment issue, or a UI change that just needs a locator repaired. Splitting the work this way isn't just an architecture preference — it's what makes the failure-diagnosis step legible enough for a human to audit, instead of one opaque agent deciding on its own that a test can be marked as passing.

The Agentic Test Loop

The cost and reliability reality check

Here's where it gets more interesting than the marketing copy suggests — and where I found the closest parallel to what I ran into researching long-context windows, where "just give the model more context" turned out to have real, measurable costs that don't show up until you look closely.

A 2026 paper on self-healing test automation makes a pointed argument: routing every broken locator through an LLM for re-discovery introduces per-run API costs that become prohibitive at enterprise regression scale — thousands of tests, run on every commit, across every browser and locale. The authors built a "zero-cost" alternative using structured accessibility-tree extraction instead of LLM calls, and found it held a substantial cost advantage over LLM-based discovery even after accounting for the engineer time it takes to maintain a more deterministic system. The lesson isn't that LLMs don't belong in the loop — it's that routing everything through a model, all the time, is rarely the efficient design.

A separate empirical study went further, examining what happens when autonomous repair agents are given a long leash in a real enterprise UI-testing environment. Working from 300 execution reports and over 600 individual test runs, the researchers catalogued recurring failure modes in high-autonomy repair loops: repairs that never converge, agents that hallucinate UI interactions that don't exist, and — more troubling — cases where an agent "fixes" a failing test by quietly weakening its assertions or deleting the check altogether, producing a false-positive pass. Their conclusion wasn't "don't use autonomous repair." It was that autonomous repair needs operational constraints — a controlled, multi-agent workflow with guardrails — rather than an agent left to loop indefinitely on its own judgment.

The pattern should sound familiar to anyone who's read the retrieval-augmented-generation literature: fully autonomous and fully manual are both the wrong defaults. The systems that actually hold up in production are the ones that route between deterministic and model-driven approaches based on the situation, not the ones that pick one and apply it everywhere.

The gap between the pilot and the pipeline

The survey data backs up what I saw anecdotally in pre-sales calls: enthusiasm for agentic testing is running well ahead of actual production deployment. The 2025–26 World Quality Report — Capgemini, Sogeti, and OpenText's annual survey of thousands of senior QA and technology leaders — found that roughly 89% of organizations are piloting or deploying generative AI within quality engineering, but only about 37% report those capabilities actually running in production. A separate 2026 enterprise testing survey found something similar from a different angle: 61% of organizations use AI across most of their testing workflows, but only 12% have handed the AI full autonomy over any part of that workflow. Everyone's experimenting. Almost nobody's letting the agent make the call unsupervised.

That gap isn't teams being slow or overly cautious — it tracks with the failure modes the arXiv research above actually documented. Gartner's broader read on agentic AI, not specific to testing, is blunter still: the firm has projected that more than 40% of agentic AI projects will be canceled before the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls as the leading causes. Every QA leader I demoed to who pushed back hard on "full autonomy" language wasn't rejecting the technology — they were, correctly, pricing in exactly this risk.

Market response: two products emerging from one shift

The commercial side of this is unbundling in a very literal way. Zoom out and the macro numbers explain why every vendor in this space is scrambling for position: Gartner projects that agentic AI will be embedded in 33% of enterprise software applications by 2028, up from less than 1% in 2024, driving more than $450 billion in associated enterprise software revenue. Testing is a small slice of that wave, but the global testing-as-a-service market on its own is projected to reach roughly $11.4 billion by 2030, growing at about a 14% CAGR, driven largely by the complexity of cloud-native and microservice architectures (Grand View Research). But that growth isn't only flowing into traditional full-service contracts.

Two distinct buying patterns are emerging from the same underlying technology shift:

Managed testing services still make sense as an anchor for organizations that want comprehensive, cross-functional coverage — security, performance, compliance, load — with predictable, contract-based cost and minimal internal QA headcount. The tradeoff is reduced visibility: you're handing over a black box and getting a report back.

Embedded agentic tooling — point products that plug directly into CI/CD and turn plain-language test intent into production-ready, auditable code — appeals to engineering-led teams that want speed, ownership, and instant feedback on every commit, and are willing to invest engineering time to get it. This is the product-led-growth end of the market, and it's where most of the "pure AI testing" startups are competing.

Webomates, where I worked, is a decent case study of a vendor straddling both sides at once. Its core offering, Webomates CQ, is a managed TaaS platform with a guaranteed execution SLA — the buyer is typically a VP of Engineering who wants coverage and predictability without growing an internal QA org. But the company also built AiScriptBuddy, a standalone tool that converts plain-English test cases into Selenium/Playwright scripts and leans on self-healing to cut locator maintenance. In pre-sales conversations, I'd notice the room change depending on which one I led with. Pitch the managed service first to an engineering-heavy audience and I'd get questions about control and pipeline integration. Open with AiScriptBuddy instead — something the team could touch, run locally, and see generate real code — and the same audience would lean in, because it read as a tool that made them more capable rather than a service that quietly took work off their plate. That's not a knock on either model; it's just a reminder that who's in the room changes which pitch lands, and increasingly, both pitches have to exist inside the same company.

Neither model is disappearing. What's changing is that they're increasingly sold, bought, and evaluated as separate decisions rather than one bundled QA contract — the same unbundling logic playing out in adjacent categories of enterprise software.

A decision framework

Distilling the above into something usable, the choice mostly comes down to four questions: How much control does the team need over execution? How fast does testing need to integrate into the existing pipeline? How broad does coverage need to be versus how deep on functional regression? And does the organization want a predictable contract or a cost that scales with its own engineering headcount?

Choosing Your QA Architecture

In practice, a lot of organizations will land on a hybrid: a managed service for the compliance-heavy, cross-functional coverage that's expensive to build in-house, paired with embedded agentic tools inside the CI/CD pipeline for the fast-moving functional regression suite that engineers want to own directly. That's the same "route by situation, not by default" conclusion that keeps showing up whenever an AI capability matures enough to have real cost and reliability tradeoffs attached to it.

Where this is heading

The interesting story here isn't really "AI is coming for QA jobs" — it's that testing is becoming another domain where the orchestration layer, not the underlying model, is the actual product. The same way retrieval systems are evolving toward adaptive routers that decide when to search versus when to reason over full context, testing platforms are evolving toward agents that decide when to act autonomously versus when to defer to a human or fall back to a deterministic check.

Several analyst firms — Gartner among them — have started projecting that agentic systems will handle the large majority of routine QA operations before the end of the decade. Whether that plays out on schedule or not, the direction is clear enough: the debate isn't scripts versus agents anymore. It's about designing systems that know which one to reach for, and when.

I didn't have this framing when I was on those calls — I was mostly just trying to figure out which demo to lead with. It's only in hindsight, and after spending the months since graduating thinking about routing decisions in a completely different part of the AI stack, that I can see the pre-sales tension I kept running into was never really about testing at all. It was the same architecture question showing up in a different domain, months before I had the vocabulary for it.

Sources