Printed Signs, Real Threats: How CHAI Hijacks Vision‑Language Models and Risks Autonomous Systems

• Executive summary
• CHAI (Command Hijacking against embodied AI) uses printed or physical signage to trick machines that read scene text—no software access required.
• Real-world tests and closed-loop simulations show high success rates across drones, vehicles, and object trackers—this converts perception capability into an attack surface.
• Immediate mitigations (text filters, cross‑modal consensus, human-in-the-loop for critical actions) and updated procurement and insurance standards are essential.

What is CHAI?

CHAI stands for Command Hijacking against embodied AI. At its core it’s simple: optimized physical signs—carefully chosen words, fonts, colors and placement—are read by vision‑language models and interpreted as instructions. Those interpretations can then influence control decisions in drones, robots, and self‑driving stacks.

Think of CHAI as the digital equivalent of a fake road sign that reroutes drivers, but the “drivers” are AI agents that can both read and act on text in the environment. No code needs to be changed and no network needs to be breached—the attack is entirely physical.

Evidence — key results and plain-English takeaways

Researchers tested CHAI across simulated and real platforms. Major highlights:

• Drone emergency landing (simulated): CHAI caused unsafe landings onto crowded roofs 68.1% of the time. Takeaway: more than two-thirds of simulated runs chose a dangerous landing spot (CHAI paper, 2026).
• Closed-loop AirSim tests: Success rose to 92% when drones interacted continuously with the environment. Takeaway: attacks are even more effective when the vehicle is in a control loop (AirSim, CHAI paper).
• DriveLM (autonomous driving): CHAI induced risky maneuvers despite pedestrian detection in 81.8% of runs. Takeaway: models can act on malicious text even when other perception signals are present.
• CloudTrack (object tracking): The highest observed success, 95.5%—for instance, mislabeling a civilian car as an emergency vehicle. Takeaway: signage can override identity classification repeatedly.
• Field tests with printed signs: Real-world experiments fooled a robot vehicle over 87% of the time across lighting, angles, and sensor noise. Takeaway: this is not just a lab artifact—physical signs work in real environments.
• Transferability and language: Attacks transferred across scenes and worked in multiple languages (Chinese, Spanish, mixed prompts), producing universal prompts that generalize. Takeaway: attackers don’t need bespoke signs for every location.
• Compared to prior methods: CHAI can be up to 10× more effective than earlier scene‑based approaches like SceneTAP. Takeaway: this is a major step up in practical attack capability.

How CHAI works (plain language)

Attackers optimize a sign in two steps: first choose the semantic prompt—the words or phrase that will nudge the model—and then tune visual features like color, font, contrast, and placement. Vision‑language models learn strong priors about how text in scenes maps to actions. If perception pipelines treat scene text the same way they treat authenticated commands, an optimized sign can flip a model from “observe” mode to “act” mode.

Transferability happens because many vision‑language models share similar priors about layout and instruction‑like text. Once an optimized prompt is found, it often works across different images, scenes, and even languages, making the attack scalable.

Business and safety implications

Allowing machines to read and act on environmental text improves flexibility and autonomy—but it also creates a new attack surface that crosses cyber and physical security. For businesses using autonomous systems, the implications are concrete:

• Perception is now part of the threat model. Treat vision‑language stacks as safety‑critical software with mandatory adversarial testing.
• Liability and insurance will need to catch up. If a malicious sign causes damage, who is responsible—the system operator, the vendor, or the model developer?
• Procurement should demand adversarial‑robustness evidence. Buying a platform without physical adversarial tests is a hidden risk.
• Regulators and standards bodies will face pressure to define minimum testing requirements and certification for embodied AI systems.

Concrete mitigations: immediate, short‑term, long‑term

Immediate (days–weeks)

• Treat scene text as untrusted input. Add pre‑response filters that flag or neutralize instruction‑like text before it influences control decisions.
• Require human confirmation for high‑risk maneuvers if scene text is involved. Human‑in‑the‑loop for critical actions reduces chance of catastrophic automation mistakes.
• Log and monitor any time scene text changes a control decision—build a forensic trail for incident response and insurance claims.

Short‑term (weeks–months)

• Require cross‑modal consensus for safety‑critical commands. If text suggests an instruction, verify it against LIDAR, radar, GPS, and other sensors before acting.
• Implement authenticated instruction channels. Only signed or verified remote commands should be acted upon without extra checks.
• Add language and font diversity checks in test suites. Ensure systems are tested with multiple languages, fonts, colors, and placements.

Long‑term (architecture & policy)

• Redesign perception pipelines so detection (what’s in the scene) is separated from decisioning (what to do). Gate text-derived instructions behind verification subroutines.
• Include adversarial physical‑world testing in vendor SLAs and procurement contracts—require resilience benchmarks and remediation timelines.
• Work with insurers and regulators to establish responsibility models and mandatory robustness testing for embodied AI.

Practical testing checklist for procurement

• Run adversarial sign tests across lighting, angles, distances, languages, and moving platforms.
• Include closed‑loop simulations (e.g., Microsoft AirSim) and physical field trials with printed signs and varied sensors.
• Ask vendors for documented mitigation measures: pre‑response filtering, cross‑modal arbitration, signed command channels.
• Require a plan for on‑site adversarial testing during acceptance and periodic revalidation after updates.

Compliance, legal, and insurance considerations

CHAI forces companies to rethink contractual risk allocation. Recommended clauses and actions:

• Require vendors to provide adversarial‑robustness test results and remediation commitments as part of procurement.
• Update maintenance and support contracts to include physical adversarial testing after major model updates.
• Engage legal and insurance teams early—policies must articulate coverage for losses caused by adversarial physical attacks and clarify vendor/operator liability.

What leaders should do next

• Audit: Map every place scene text can influence a control decision and record vendor claims about adversarial robustness.
• Require: Proof of adversarial testing and mitigation from suppliers before deployment or upgrade.
• Govern: Create an “AI safety” sign‑off for high‑stakes autonomous deployments and budget for regular cross‑modal testing.
• Coordinate: Bring operations, security, legal, procurement and insurance together to design a response and update SLAs.

Quick checklist for AI ops and security teams

• Immediate: Add a pre‑response filter that strips or flags command‑like scene text.
• Short: Implement cross‑modal overrides (radar/LIDAR) for text‑triggered actions.
• Long: Redesign pipelines to separate detection from decision and require signed command channels for autonomous maneuvers.

“New technologies bring fresh vulnerabilities; researchers must anticipate misuse and design defenses before exploitation,” — Alvaro Cardenas, Professor of Computer Science, UC Santa Cruz.

“We can build attacks that work in the physical world; new defenses are required,” — Luis Burbano, PhD student and first author of the CHAI study (arXiv/2026).

Limitations and what CHAI does not (yet) prove

The experiments cover a set of models, platforms, and scenarios; they do not prove every vision‑language stack is equally vulnerable. Success rates vary by model, sensor suite, and deployment details. The tests do, however, demonstrate a clear and practical attack vector that operators must treat seriously. Expect variance in real deployments and plan defenses accordingly.

For engineers — tactical steps

• Instrument perception outputs: tag every text detection event and the downstream decision it influenced.
• Implement a “safety oracle” that denies text-derived commands unless corroborated by independent sensors or human approval.
• Build automated adversarial test harnesses that cycle fonts, colors, and languages against your perception pipeline.

For executives — strategic steps

• Require adversarial physical‑world testing in vendor contracts and acceptance criteria.
• Mandate cross‑functional governance—security, engineering, legal and procurement must jointly own embodied AI risk.
• Allocate budget for periodic revalidation following model updates and for insurance coverages that reflect adversarial risks.

CHAI turns a helpful capability—letting machines read their environment—into a practical exploit. The vulnerability connects model alignment, perception engineering, and physical security. The pragmatic response is straightforward: treat multimodal perception as part of the attack surface, add layered mitigations now, and require vendors to prove resilience as a condition of deployment. Doing so protects people, reputations, and the business value AI is supposed to unlock.

Suggested visuals & alt text:

• Annotated diagram of attack flow — alt: “Sign being read by vision‑language model, producing a control command.”
• Table comparing CHAI vs. SceneTAP — alt: “Comparison of CHAI and prior scene‑based attacks showing success rates.”
• Checklist infographic — alt: “Quick mitigation checklist for executives and engineers.”