A wall of green checks is a miserable way to learn that nobody owns the tests. Vinicius Pedroni and Jan Wiemers, both Senior Solutions Architects at AWS, are writing a series on QA Studio, a reference design customers deploy themselves for quality-assurance automation with Amazon Nova Act. The first part covered tests written in natural language and carried out by an AI agent that visually navigates the application. The second added regression suites and a command-line path for batch runs in continuous integration and delivery. This third installment starts after the automation exists. It is about grouping the tests, reading their health, and deciding who may see or change them.
When writing the test is no longer the hard part
Their claim is direct. Once many squads have automated tests, writing more of them is no longer what slows the organization down. Sorting those tests, and controlling who can touch them, becomes the work. Without an owner, nobody is clearly answerable when coverage thins out or a suite starts looking sick.
That is also how a lot of AI automation shows up in a company, in my experience. A pattern, not a law. The demo is an agent walking a flow. The later problem is hundreds of tests across products and squads, and no quick answer to a managerial question: which team owns this application and its tests, and who answers when coverage slips?
An application is a product with a pulse
In this design, an application is the container for a system under test: a website, a web app, or a service front end. You create one with a name and a base URL, plus an optional set of environments. Development, staging, and production are the examples they use. Individual tests, called use cases, and regression collections, called test suites, belong to that application and inherit its ownership. You do not assign an owner to every click-path.
Every run is written as Amazon CloudWatch custom metrics in a namespace named QAStudio/Executions, broken out by application and by environment. The application page leads with three figures for the window you select: pass rate, total executions, and the current failure count. Charts cover executions, successes, failures, and average duration, and you can filter to one environment so a calm staging run cannot pass itself off as production. A 7-day view and a 30-day view are both available.
Flaky tests (use cases that flip between passing and failing) are ranked over the last seven days. Recent failures link to the agent’s trajectory logs and session recordings, so a red result is something a person can watch, not only a counter.
The health score runs from 0 to 100. It starts from the pass rate and subtracts a penalty for flaky tests, so a suite that passes while oscillating does not look spotless. The size of that penalty, and exactly what counts as a flip, is not published. Neither is any cutoff for healthy, and no sample pass rates or execution counts are given. Ask for the formula before anyone treats the number as a KPI or puts it in a board deck. Until then it is a signal with unpublished math, not an audited score.
Five roles, and a ceiling that does not rise
Each application belongs to exactly one team. Members of that team can work with it. Everyone else does not see it, unless the owning team shares it, or unless the person is a studio-wide Admin, who can see every team, application, and user. Roles are cumulative. Each includes the permissions of the roles beneath it.
- Viewer has read-only access to the team’s applications and results.
- Executor can run use cases and suites, and cannot modify them.
- Editor creates and edits use cases and test suites inside the team’s applications.
- Team Lead manages that team’s members and creates applications for the team.
- Admin is not a badge you assign inside one team. It covers creating teams and assigning applications, along with everything the other roles can do, across the studio.
Sharing comes in three grant levels: view, execute, or full. The rule that matters is the ceiling. Effective permission on a shared application is the lower of the person’s own role and the grant. Offer a Viewer full access and they stay view-only. An owning team can open an application widely without handing a recipient more authority than that person’s role already allows. What “full” includes, relative to Editor or Team Lead, is not spelled out beyond that ceiling.
The reference deployment they show is small on purpose: four users across two teams (two Team Leads, one Viewer, and one studio-wide administrator). Treat that headcount, and the demonstration team labels in the screenshots, as an illustration of the model. It is not a picture of how any real organization staffs QA, and it is not a usage statistic.
The ceiling rule is shown with a separate walkthrough, not as a fifth person in that cast. One team grants another view access to an application. An Editor on the receiving team still sees it read-only, because the grant sits under their role. That is the point.
What a security review will ask
Before the service names, the question is plain: who may see this application, who may change a test, and where the records live.
As described, each team maps to a set of Amazon Cognito groups, one per role. An AWS Lambda authorizer turns those groups into API scopes and team context on every request through Amazon API Gateway. Teams, memberships, application ownership, and cross-team share grants sit in an existing DynamoDB single-table design. The design checks access twice: whether the caller’s role allows the action, and whether the caller’s team owns the application or has been shared with it.
A pipeline does not have to be a person at a keyboard. The design can reuse the OAuth client-credentials flow from the earlier part of the series, so a job authenticates as a machine identity with only the scopes you grant it. That makes it possible to keep the identity that runs a suite from being the identity that rewrites it. It does not do the split for you. If the client is issued editor scopes, the runner is an editor. Leaving that choice until after an agent can already reach a production URL is, in my view, the version you do not want to explain in a security review.
Customers deploy the reference solution into their own AWS account with the AWS Cloud Development Kit. The write-up says test data, results, and metrics stay inside that account. For anyone whose review opens with where this leaves our tenant, that boundary is the feature. Recent failures point at Nova Act trajectory logs and session recordings, but the post does not name a separate store for those recordings. Confirm where they are written before you assume they are only CloudWatch metrics.
What this means if you are betting on AI agents
The fashionable version of AI for business says the agent will author the tests, run the tests, and file the bug, and the QA org will shrink to a rounding error. The less fashionable version is the one this design is built for. AI agents multiply tests. They do not multiply owners. Team ownership is a place to put accountability. It is not, by itself, a named person on the hook.
Picture a production run that is entirely green: the right environment, a pass rate that looks fine, no failures in the window, and no team that owns the application. You have a clean pipeline and nobody to call on Monday. That is how coverage slip becomes an incident story. Inside this reference design, an application has to belong to one team, which is the handle. Whether someone on that team has agreed to act when the score moves is still an operating decision.
There is a useful dullness in separating Executor from Editor, and in refusing to let a share outrank someone’s role. Least privilege feels like paperwork until an AI agent holds a token that can click through production. The ceiling rule is dull in exactly the right way.
Two limits, stated plainly. This is a reference architecture customers stand up themselves, not a managed product with a published price, a concurrency guarantee, or an auditable health formula in this write-up. And agentic here means visual navigation with Amazon Nova Act. It is not Amazon Nova 2 Sonic, AWS’s separate work on real-time conversational audio. Same brand family, different job. Do not let the name do the explaining.
If you are piloting AI agents against live applications, settle the owning team, the edit-versus-run split, and the environment boundary before you celebrate the pass rate. A score that marks down flaky tests only helps if someone has agreed to look when it moves, and if you know how the number is built.
Questions worth settling first
- What does grouping tests into an application actually fix?
It gives a system under test a name, a base URL, optional environments, and a single owning team. Use cases and suites inherit that ownership, so health sits at the product level instead of floating across hundreds of unrelated tests. A team is not the same thing as a named person who will act.
- How should I read the 0-100 health score?
It starts from pass rate and subtracts a penalty for flaky tests, so a suite that passes while flipping does not look perfect. The penalty formula, the definition of a flip, and any healthy cutoff are unpublished. Ask for the math before you manage to the number.
- Who can change a test, and who can only run it?
Editors create and edit use cases and suites. Executors run them. Viewers only read. Team Leads manage members and create applications. Admin is studio-wide and can see applications that have not been shared. Each role includes the permissions beneath it.
- Can a share grant give another team more power than their role allows?
No. Effective access is the lower of the person’s role and the grant (view, execute, or full). A Viewer offered full access stays view-only. The write-up does not define what full adds beyond that ceiling.
- Where do results, access records, and session recordings live?
You deploy with CDK into your own AWS account. Execution outcomes are CloudWatch metrics; teams, memberships, ownership, and share grants are in DynamoDB; identity is handled through Cognito. The post says test data, results, and metrics stay in that account, and recent failures link to trajectory logs and session recordings. It does not name a separate store for the recordings, so confirm that path in the deployment.
- Does a CI pipeline need a person to log in?
Not in this design. It can authenticate with OAuth client credentials as a machine identity limited to the scopes you issue. That enables a runner that is not also an editor. It does not guarantee the split unless you actually withhold edit scopes.
- What has to be settled before a pilot touches production?
The owning team, who may edit versus who may only run, which environment the agent may reach, and whether the pipeline’s machine identity is limited to run scopes. If anyone will manage to the health score, get the flaky-test formula first. Confirm where session recordings are written before you treat the account boundary as fully mapped.