How ZDNET Tests AI: A Hands‑On Framework for Evaluating LLMs, AI Agents, and AI Automation

• Executive summary
  • Every review starts with real-world use: vendor claims and press-release numbers don’t replace hands-on testing for LLM testing, AI agents, or AI automation tools.
  • Two complementary tracks deliver value: standardized comparative reviews for apples‑to‑apples scoring, and long‑form “living with” projects that surface real integration, safety and productivity realities.
  • Neutrality, reproducibility and periodic re-testing (typically every 6–12 months) are built into the process so businesses can make procurement decisions with less risk.

The single rule that drives every test

When vendors package benchmarks into marketing, buyers need a reality check. That reality check is simple: each product is evaluated through direct, real‑world use. Hands‑on AI reviews cover large language models (LLMs), coding assistants, image generators, AI website builders, AI agents and broader AI‑enabled apps. Vendor demos and access help with logistics, but they are never substitutes for independent testing.

Every review must be based on direct hands‑on, real‑world testing rather than vendor claims or press‑release numbers.

Two testing tracks: Comparative reviews and living‑with projects

There are two complementary ways to test AI so that business buyers get both repeatability and realism.

1. Comparative AI reviews — repeatable and auditable

Comparative reviews are built so procurement teams can compare contenders objectively. The workflow is intentionally methodical:

1. Define evaluation criteria and concrete tests (accuracy, latency, cost, integration effort, safety checks, privacy controls).
2. Shortlist 5–10 candidates based on market leaders (ChatGPT, Gemini, Claude), reader tips, social signals and vendor submissions that actually fit the category.
3. Run standardized, repeatable tests for each criterion with the same inputs and datasets.
4. Normalize scores so different test scales become comparable, then apply weighted metrics to reflect buyer priorities.
5. Document the method, publish the test descriptions and update periodically as tools change.

This structure turns subjective impressions into a reproducible scorecard that teams can reuse when evaluating AI for business or AI automation projects.

Sample scoring rubric (template)

• Accuracy / Correctness — 30%: factual correctness, hallucination rate under test prompts.
• Safety & Privacy — 20%: red‑team prompt resilience, data handling, PII/PHI leakage checks.
• Performance & Latency — 10%: response time at realistic concurrency levels.
• Integration & Developer Experience — 15%: SDKs, APIs, docs, onboarding friction.
• Value & Cost — 15%: cost per useful output, pricing predictability.
• Reliability & Versioning — 10%: uptime, model-change transparency, update cadence.

How normalization works: each raw test returns a score on its native scale; those scores are scaled to a 0–100 band and multiplied by the metric weight. The weighted numbers produce a single composite score you can rank. For transparency, publish raw scores, normalization method and the weight table so buyers can adjust weights for enterprise vs. SMB priorities.

2. Long‑form “living with” projects — messy reality, clearer signals

Short tests catch surface‑level capability. Sustained projects catch integration costs, safety gaps and workflow changes. Long‑form experiments involve building real things: prototypes, production features, or week‑to‑week operational usage.

Examples of what living‑with tests reveal:

• Productivity lifts at prototyping stage — one coding experiment compressed a prototyping phase often described as 24 days of coding into roughly 12 hours of focused interaction with an AI coding assistant. That doesn’t eliminate later debugging and integration work, but it changed the shape of iteration.
• Rapid ideation at product level — a staged experiment claimed that early concept work could be compressed dramatically (examples discuss compressing stages of product development from years into days), showing the tools’ power for front‑loaded creative and requirements work.
• New failure modes — hallucinations, brittle prompts, and API edge cases often only surface after several deployment cycles or when the system sees real user inputs.

That combination of dramatic productivity gains and subtle new failure modes is why long‑form testing complements comparative reviews: they answer different procurement questions.

Safety, privacy and reproducibility — testable, not just checkboxes

Safety and privacy are not add‑ons. They’re core test categories with concrete checks:

• Red‑team prompts to test for malicious or harmful content generation.
• Data‑leak tests using seeded PII/PHI in training/test scenarios to detect leakage in outputs and logs.
• Threat‑model assessments: what happens if the model is given internal documents, customer lists, or regulated data?
• Auditability checks: are logs available, are model weights/version timestamps published, and can customers get exportable evidence for compliance?

Publishing the test harness makes reproducibility possible. Where practical, share:

• Prompts and input data (sanitized),
• Evaluation scripts or scoring rules,
• Baseline seed outputs and expected behavior notes,
• Instructions to run the same tests locally or in a sandbox.

Enterprise‑only or early‑access models complicate reproducibility. For those, practical options include vendor‑supplied test instances under an NDA, staged sandbox access, or federated test harnesses that run locally against customer data without sharing it externally.

Operational realities: access, logistics and editorial independence

Testing AI is operationally messy. Candidate selection blends market visibility (ChatGPT, Gemini, Claude), community signals, reader requests and vendor submissions. But access is rarely instantaneous: coordinating logins, quotas, API keys and consistent datasets can take weeks or months. A first review of AI website builders required 231 emails and more than six months to arrange; subsequent updates took far less time as processes matured.

Editorial independence matters. Vendors may provide access to tools, early‑access keys or documentation, but they do not get pre‑publication visibility or editorial control. That separation preserves trust for readers and reduces conflict between advertising/sponsorship relationships and reviews.

How to evaluate an AI agent for a business use case (mini‑test: AI for sales)

Here’s a concise, practical test you can run when evaluating an AI agent intended for lead qualification or sales automation.

1. Set KPIs: lead conversion rate, false positive rate (bad leads accepted), time‑to‑qualification, and handover accuracy to human reps.
2. Prepare data: a representative sample of inbound leads, historical labels (qualified/unqualified), and a set of edge cases that historically confused reps.
3. Design prompts and integration tests: simulate real queries from your CRM, include human‑in‑the‑loop handoff scenarios, and test rate‑limits/concurrency.
4. Measure privacy risk: ensure the agent won’t exfiltrate sensitive fields and that transcripts are stored per your data retention policy.
5. Run A/B tests: live test with a control group and measure lift in conversion, reduction in rep time, and unexpected regressions (e.g., increased follow‑ups due to hallucinated lead details).
6. Post‑test governance: inspect failure cases, produce a risk log, and require a remediation plan before production rollout.

This same structure adapts to other verticals — AI for customer service, AI for marketing, or AI for software development — with the KPIs customized to the domain.

Practical checklist for procurement teams

• Require real‑world demos using your data or a sanitized representative dataset.
• Ask for a published test methodology, sample prompts, and scoring rubrics.
• Insist on contract terms covering data residency, audit rights, model‑change notifications, and rollback paths.
• Budget for re‑evaluation: plan for a re‑test cadence of 6–12 months, since model updates can change outcomes quickly.
• Include safety and privacy tests in RFP scoring, not as afterthoughts.

Key questions and answers

• How are candidates chosen?

  By market prominence (e.g., ChatGPT, Gemini, Claude), reader requests, social/industry signals and vendor submissions that fit the testing constraints.

• Can vendors influence reviews?

  Vendors can supply access and documentation, but they receive no pre‑publication visibility and have no editorial control—maintaining neutrality between access and influence.

• Why run long‑form experiential testing?

  Short prompts expose capability; long‑form projects expose integration costs, governance needs, hallucination patterns and real productivity impacts across weeks or months.

• How often should winners be re‑tested?

  Regularly—most “best” lists can change within six to twelve months as features, models and pricing evolve.

Who should care — and what to do next

For the C‑suite: Treat AI adoption as a lifecycle decision, not a one‑time purchase. Require hands‑on validation, contract clauses for data residency and update notifications, and budget for periodic re‑evaluation.

For procurement: Embed safety, privacy and reproducibility checks into your RFPs. Use the sample rubric above and make weights configurable for different business priorities.

For engineers and product teams: Run living‑with experiments before production. Prototype with an AI agent in a sandbox to reveal hidden integration costs, debug workflows, and measure the human‑in‑the‑loop handoff points.

Testing AI responsibly is no longer an academic exercise; it’s a procurement and product imperative. Insist on hands‑on validation, documented methodology and periodic re‑testing before you commit critical workflows to any AI agent or LLM.