Visual agentic UX testing with Amazon Nova Act and Bedrock: scalable pattern and pilot plan

TL;DR

Visual, agentic testing, using Amazon Nova Act agents together with generative flow discovery, provides a practical way to scale UX testing across many user journeys while reducing selector maintenance. The AWS reference pattern ties Nova Act, Amazon Bedrock (for flow generation and analysis), and standard AWS building blocks (S3, Lambda, DynamoDB, ECS/Fargate) to generate, execute, log, and analyze flows. Run a short pilot (3-5 flows) and measure success rate, time per flow, and cost per flow before you expand.

The problem at scale

Manual UX review finds nuance but does not scale. Scripted automation (Selenium, Playwright) reduces manual effort, but it often breaks when dynamic single page apps change DOMs or CSS classes. An alternative is visual, agentic testing: agents interpret screenshots and page layout to decide actions and record their reasoning. That mirrors how a human navigates a UI, but it introduces model-specific failure modes like hallucination or inconsistent behavior.

What the pattern does

At a high level the pattern:

Ingests documentation and site artifacts into S3 and a Bedrock Knowledge Base for semantic search.
Uses a generative model (the sample uses Claude 4.5 Sonnet via Amazon Bedrock, confirm availability in your account and region) to convert tasks and docs into test flows at multiple granularity levels.
Orchestrates parallel runs with AWS Lambda and ECS/Fargate, executing Nova Act agents that interact with headless browsers visually.
Collects screenshots, chain-of-thought logs, and behavioral traces in S3, then runs post-processing (Bedrock and Lambda) to compute usability metrics surfaced in a React dashboard.

Four tidy architectural layers

Documentation processing layer: S3 and Bedrock Knowledge Base for semantic similarity search over product docs and support articles.
Orchestration layer: Flow discovery Lambda that calls a Bedrock LLM for generation, DynamoDB to store flows, and DynamoDB Streams to trigger execution.
Execution layer: Nova Act agents run in parallel on ECS/Fargate, driving headless browsers, taking screenshots, and emitting chain-of-thought logs.
Analysis layer: Post-run Lambdas call Bedrock to compute error-adjusted metrics and timing by granularity. Results are written to DynamoDB and shown in a React dashboard for human triage.

Key implementation artifacts to review

Sample repository: aws-samples/sample-nova-act-ux-testing (contains the reference implementation and deployment instructions).
Nova Act SDK: https://github.com/aws/nova-act, the SDK and README are useful for SDK-level primitives and recommended usage patterns.
Example execution JSON (valid JSON snippet):

{
  "flow_id": "ecommerce_purchase_flow",
  "batch_id": "batch_12345",
  "timestamp": "2026-07-01T20:50:01.811454",
  "steps": [
    "search for desk lamp",
    "select the first result"
  ]
}

Note: the sample references model calls to Claude 4.5 Sonnet via Amazon Bedrock for flow discovery and analysis. Confirm that model is available in your target region and account before relying on it.

How flows are generated and run

Flow discovery: documentation and support text are pushed into S3 and indexed in an Amazon Bedrock Knowledge Base. A flow discovery Lambda calls a Bedrock LLM (the sample uses Claude 4.5 Sonnet) to transform high-level tasks into flows at three granularities (high-level goals → mid-level subgoals → step-by-step instructions), then writes the flows into DynamoDB.

Orchestration & execution: DynamoDB Streams trigger a coordinator Lambda that queues batches and spins up ECS/Fargate tasks. Each task runs an Amazon Nova Act agent in a headless browser session. Agents log interactions, take screenshots, and write chain-of-thought files and artifacts to an S3 results bucket.

Analysis: a post-processing Lambda can call Bedrock to compute error-adjusted signals, timing by granularity level, and friction hotspots. Summaries are written to DynamoDB and shown in a React dashboard with Overview and Performance tabs.

Why this approach helps, and where it doesn’t

What it helps with:

Reducing selector churn. Agents use visual and semantic cues rather than brittle DOM selectors, so many UI changes do not require script rewrites.
Exploratory coverage. Generative flow discovery can surface journeys engineers had not scripted manually.
Faster triage. Screenshots plus chain-of-thought logs give human-readable debugging context for failures.

Where to be cautious:

Non-determinism. Model-driven agents can behave inconsistently across runs. Do not rely on them for legally or financially critical assertions without deterministic checks.
Anti-bot protections. CAPTCHAs, 2FA, aggressive rate limiting, or bot detection can block agents. Use test accounts, dedicated SSO test endpoints, or human-in-the-loop steps where necessary.
Privacy risk. Screenshots and chain-of-thought logs may contain PII. Implement redaction, encryption, and short retention windows.
Model and region availability. Bedrock models and specific partner models vary by region and account access. Confirm availability before making design decisions.

Concrete failure modes and mitigations

Agent clicks wrong element because labels are visually ambiguous. Mitigation: add text-anchoring assertions that verify visible text near the click target, or add a DOM snapshot assertion. Fall back to backend API checks if possible.
Flow fails on MFA or CAPTCHA prompts. Mitigation: use dedicated test SSO endpoints, persistent browser sessions for test accounts, or human-in-the-loop handoffs for MFA steps.
Rate limiting or IP blocks during parallel runs. Mitigation: stagger concurrency, use pools of test IPs, and implement exponential backoff and retry logic.
PII captured in logs or screenshots. Mitigation: redact artifacts before long-term storage, apply S3 SSE-KMS encryption, enforce least-privilege IAM, and use object tagging plus lifecycle policies to expire sensitive artifacts.
Model drift or inconsistent outputs across UI changes. Mitigation: version prompts and flows, break long flows into staged runs, and re-generate flows after major UI releases.

Operational recommendations

Split prerequisites. Nova Act execution is Python centric. Install with pip install nova-act and run on Python 3.10 or newer. The React dashboard requires Node.js v20 and npm v10.8 or newer. Keep execution (Python) and front-end (Node) environments separated.
Use dedicated test accounts and test SSO endpoints. Store credentials in Secrets Manager and mount ephemeral browser profiles for isolated authenticated sessions.
Limit flow length. According to the Nova Act SDK README, tasks work best when kept to under about 30 sequential steps. Break complex journeys into stages for better reliability.
Capture objective artifacts with model logs. Always include screenshots, DOM snapshots, and backend traces where possible. Treat chain-of-thought logs as heuristic debugging aids, not legal evidence.
Security and governance: encrypt S3 at rest with SSE-KMS, restrict IAM roles, apply lifecycle policies, and implement an automated redaction step (simple regex or a PII detection API) before storing artifacts long term.
Accessibility (a11y): run Axe or Lighthouse in parallel and record accessibility violations with each flow result.

Pilot plan and measurable metrics

Before you expand, run a small pilot and collect three metrics:

Success rate = successful steps ÷ total attempted steps (per flow and aggregated).
Time-per-flow = average runtime per flow (wall time from start to termination).
Cost-per-flow = (Bedrock_inference_minutes × Bedrock_cost_per_min + NovaAct_runtime_minutes × NovaAct_cost_per_min + ECS/Fargate_compute_cost + S3_storage_and_IO + Lambda_execution_cost) ÷ number_of_flows.

Suggested experiment:

Select 3-5 representative flows (checkout, search → filter → buy, account settings).
Measure a baseline using your existing selector-based automation: success rate, time, cost.
Run the Nova Act approach at low concurrency, measure the same metrics, and inspect screenshots and chain-of-thought logs.
Introduce a targeted UI change (reorder elements or rename classes) and re-run both approaches to measure resilience.

Decision heuristic: if Nova Act gives you greater than 85% success rate on critical flows, reduces time-to-triage by 20% or more, and cost-per-flow is within an acceptable multiple (for example, less than 3× your deterministic tests while providing new exploratory coverage), consider a staged rollout with a mixed strategy of agentic and selector-based tests.

Quick validation checklist

Confirm model availability: verify the Bedrock model used for flow generation and analysis (for example, Claude 4.5 Sonnet) is available in your AWS account and region.
Run a small batch: execute three representative flows, inspect S3 artifacts (screenshots, chain-of-thought), and compute success rate, time, and cost.
Verify PII handling: ensure redaction or masking pipelines are in place and S3 encryption and IAM policies restrict access.

Where to look in the sample

Repository: aws-samples/sample-nova-act-ux-testing (clone via HTTPS):
git clone https://github.com/aws-samples/sample-nova-act-ux-testing.git
cd sample-nova-act-ux-testing
cp template.env .env
./deploy.sh
Cleanup: cdk destroy
Key file paths referenced in the sample: lambda/flow_discovery/index.py and ecs/ecs_act_headless/app.py (use these as starting points to confirm Bedrock model calls and Nova Act invocation).
Nova Act SDK: https://github.com/aws/nova-act, review README for SDK usage patterns, API keys and IAM auth, and the <30 step guidance.

Selected example prompts and strings (preserved from sample)

“purchase a highly-rated stainless steel coffee maker”
“buy a stainless steel toaster with good ratings via the search”
“search for desk lamp”
“select the first result”
“search for bananas”
“purchase 2 bunches”
Example persistent session starting page: “https://yourapp.com/purchase”

Key takeaways, questions you should ask

Can Nova Act replace selector-based automation for all tests?

Not entirely. Use visual, agentic testing for exploratory coverage and churn-prone UI; retain selector-based tests for deterministic, high-stakes transactions (checkout, billing). A hybrid strategy is usually most practical.
How do I handle authentication and MFA in automated runs?

Prefer dedicated test SSO endpoints and test accounts. Persistent browser sessions (example: NovaAct(starting_page="https://yourapp.com/purchase", user_data_dir="/tmp/authenticated-session", clone_user_data_dir=False)) help for session persistence, but for MFA consider human-in-the-loop handoffs or test SSO that bypasses MFA for automation.
Will this reduce maintenance time?

It can significantly cut selector maintenance for churny UIs, but you’ll trade maintenance of prompts, flow specs, and monitoring for model drift. Plan ops for prompt/version management and periodic revalidation.
What about cost and scale?

Costs vary with Bedrock model calls, Nova Act runtime, ECS/Fargate compute, and S3 usage. Start small, measure inference minutes and compute minutes, and use that to extrapolate to your expected concurrency and run cadence.
How trustworthy are chain-of-thought logs?

Chain-of-thought logs are useful debugging artifacts but are not authoritative. They may be incomplete or misleading, always pair them with screenshots, DOM snapshots, or backend traces before acting.

Next practical steps

Clone the sample repo and confirm prerequisites for both execution (Python 3.10+, pip install nova-act) and the dashboard (Node.js v20+, npm v10.8+).
Run a 3-5 flow pilot, collect success rate, time, and cost, and inspect artifacts in S3 for PII exposure and signal quality.
Introduce a small UI change to test resilience versus selector-based automation, then decide whether to expand agentic tests under a mixed strategy.

Visual agents are not a silver bullet, but when combined with semantic flow discovery and automated analysis they provide an efficient way to broaden UX coverage, reduce brittle selector maintenance, and surface why users get stuck. Treat them as a complementary layer in your testing portfolio, with clear pilots, security controls, and objective metrics guiding any rollout.