AI Security - Custom Font Rendering Can Poison Systems

What Happened

A new attack technique has emerged that targets AI web assistants like ChatGPT and Claude. This method exploits a fundamental flaw in how these systems interpret web content. By using a custom font file and basic CSS, attackers can deliver malicious instructions without detection. The attack was demonstrated by LayerX, who created a fake webpage that appeared harmless but contained hidden threats.

The technique takes advantage of the gap between what a browser renders visually and what an AI tool reads from the underlying HTML. When AI assistants analyze a webpage, they rely on the raw HTML structure. However, the browser interprets this structure through a visual pipeline, which can lead to discrepancies. This disconnect allows attackers to manipulate the AI's perception of the content, making it seem safe when it is not.

Who's Affected

The attack impacts several popular AI assistants, including ChatGPT, Claude, and Gemini. In tests, these systems failed to detect the malicious content, often encouraging users to follow harmful instructions. This highlights a significant vulnerability in AI security protocols, as users trust these tools to provide safe browsing experiences.

The implications are serious, especially in environments where AI tools are integrated into workflows. The potential for AI-assisted social engineering attacks increases, as malicious actors can exploit the trusted reputation of AI systems to manipulate users effectively. This could lead to unauthorized access to sensitive information or systems.

What Data Was Exposed

While the immediate risk involves social engineering, the underlying attack method can expose users to various threats. The hidden payload in the custom font can instruct users to execute harmful commands, such as running a reverse shell on their machines. This could result in data breaches or unauthorized access to personal and corporate systems.

LayerX's proof-of-concept demonstrated how easily this attack could be executed without any JavaScript or browser vulnerabilities. The flaw lies in the AI's inability to recognize that the visual representation of a webpage can differ significantly from its underlying HTML content.

What You Should Do

To mitigate this risk, AI vendors must adopt better security practices. LayerX recommends implementing dual-mode render-and-diff analysis to compare rendered content with the underlying HTML. Additionally, AI systems should treat custom fonts as potential threats and scan for CSS techniques that hide content.

Users should also be cautious when following instructions from AI assistants, especially if they involve executing commands on their devices. Awareness of this vulnerability can help users make informed decisions and avoid falling victim to such attacks. As AI continues to evolve, staying vigilant about these emerging threats is essential for maintaining security.