Prompt Injection - Security Risks of Generative AI in Government

What Happened

Generative AI (GenAI) has become a routine part of daily operations in state and territorial governments. This shift introduces new security risks, particularly from a technique known as prompt injection. A recent report from the Center for Internet Security (CIS) highlights this persistent threat as governments increasingly rely on AI tools.

Adoption Expands Exposure

The use of AI tools among government IT teams has surged. A 2025 NASCIO survey revealed that 82% of state and territorial CIOs reported employees using GenAI in their daily tasks, a significant increase from 53% the previous year. These tools assist with various tasks, including summarizing documents, responding to emails, writing code, and managing schedules. However, their privileged access to sensitive systems makes them attractive targets for cybercriminals.

Model Behavior Creates a Security Gap

Prompt injection has been a known issue for over a decade, with roots tracing back to 2013. The problem arises from how language models process input. They do not distinguish between normal requests and malicious instructions. This vulnerability allows for two types of prompt injection:

• Direct Prompt Injection: Occurs through direct interaction with the model, attempting to override its safeguards.
• Indirect Prompt Injection: Involves embedding malicious instructions within external content, such as web pages or emails, which the AI later processes.

The report warns that prompt injections can poison GenAI databases, allowing attacks to persist across user sessions and affect other applications.

Examples Show How Attacks Unfold

Several proof-of-concept scenarios illustrate the potential for prompt injection to compromise AI systems:

• An AI agent scanning a webpage can be misled by hidden instructions in the content, leading to sensitive data being transmitted externally.
• In one case, a GenAI code assistant processed malicious instructions from a documentation page, inadvertently sending sensitive AWS API keys to an external URL.
• Another incident involved an update to the Amazon Q extension for Visual Studio Code, which included a prompt that could delete files and terminate AWS servers. AWS issued a patch shortly after.

Controls Focus on Limiting Access and Oversight

To mitigate these risks, organizations should:

• Define acceptable use policies for AI tools.
• Provide user training on handling sensitive data and recognizing malicious prompts.
• Monitor which systems and data AI platforms can access, enforcing a least privilege policy.
• Require human approval for actions involving sensitive data or code execution.
• Conduct regular log reviews to identify unusual behavior.

By implementing these measures, organizations can better protect themselves against the evolving threats posed by prompt injection in generative AI applications.