Prompt Injection Attacks: How to Test LLM Apps and AI Agents

Prompt Injection Attacks: How to Test LLM Apps and AI Agents

Prompt injection happens when untrusted language changes what an AI system does. The important target is not a chatbot saying an unusual sentence. It is the trust boundary between model output and private data, tools, accounts, or actions.

This guide focuses on authorized testing with measurable impact. Do not send destructive tool calls, access another person's data, or test outside the program scope.

Direct vs indirect prompt injection

Direct injection is entered by the user in a chat, form, or API request. Indirect injection is hidden in content the model later reads: a web page, support ticket, document, email, repository, or retrieval result.

Indirect injection is often more valuable because the victim does not intentionally submit the attack. An agent may browse attacker-controlled content and treat embedded text as a new instruction.

Map the real attack surface

Before testing prompts, document the system around the model:

• What private context can it read?
• Which tools can it call?
• Can it send messages, edit files, run code, or make purchases?
• Is memory shared across users or sessions?
• Which external documents enter retrieval?
• Does a human approve sensitive operations?

The model is one component. Authorization failures usually occur in the surrounding application.

A safe testing workflow

Start with a harmless marker. Ask the system to return a unique string only if it followed an instruction from untrusted content. Repeat the test through each input channel: user chat, uploaded files, retrieved pages, email, and tool output.

Next, test instruction priority. Can content override the task, reveal hidden context, or change the requested output format? Record the exact model, feature, account role, and conversation state because probabilistic behavior must be reproducible across several attempts.

Then test boundaries without collecting secrets. Use a test account and seeded canary such as BF_TEST_SECRET_92. A report that proves access to your own canary is safer and clearer than exposing real customer data.

Finally, test tools with a non-destructive action. For example, attempt to draft—but not send—a message, request access to an out-of-scope object, or call a tool with an invalid test identifier. The application should enforce permission checks independently of the model's decision.

What counts as meaningful impact?

A strong finding demonstrates one of these outcomes:

1. Cross-user or cross-workspace data exposure.
2. Unauthorized tool execution.
3. Persistent instruction stored in memory.
4. Secret leakage from connected systems.
5. A reliable integrity change, such as modifying a test record.

System-prompt text by itself is often informational unless it contains credentials or enables another boundary bypass.

Common false positives

Role-play, safety-policy bypasses, and strange model output are not automatically product vulnerabilities. A prompt that succeeds once in twenty attempts may also be too unreliable unless the impact is severe. Separate model behavior from application authorization and explain why deterministic controls failed.

How to write the report

Include the trust boundary, attacker-controlled input, victim action, exact prompts, repeatability rate, connected tools, canary evidence, and minimum privileges required. Add a short attack diagram and redact sensitive output.

Defensive checklist

• Treat user input, retrieved content, and tool responses as untrusted.
• Separate instructions from data with structured formats.
• Validate model output before it reaches code or APIs.
• Give every agent and tool the least possible privilege.
• Require human approval for high-impact actions.
• Isolate memory by user, tenant, and session.
• Log tool decisions and test adversarial cases continuously.

Prompt filters can reduce common attacks, but they are not an authorization layer. The strongest fix limits what a compromised model can access and do.

Original Bugflare guide based on OWASP's prompt injection prevention guidance and AI-agent security recommendations.