LLM Security: Threats, Testing Methods, and Defensive Controls

LLM security is not only a model problem. A large language model feature can expose data, invoke tools, or influence business decisions through the surrounding application. Teams shipping chat assistants, retrieval systems, and agentic workflows need to assess the model and the paths connected to it. This guide explains common threats, practical testing boundaries, and controls that reduce risk without treating model behavior as fully predictable.

Explore agentic AI penetration testing

What does LLM security cover?

LLM security covers the ways an AI-enabled application can be manipulated, misused, or made to reveal information. The model is one component. The full system may also include user interfaces, system instructions, retrieval sources, APIs, identity controls, plugins, databases, and external tools.

That distinction matters. A model may produce a safe answer in a direct chat test, yet the application can still have a weakness when it retrieves a private document or passes model output to a privileged tool. Security testing should therefore follow data and actions through the complete application, from input to final effect.

For a broader view of how testing fits into a security program, see this guide to security assurance verification. For teams already building API-backed AI features, the practical guidance on API security testing is also relevant.

Which threats should an LLM security review consider?

Threats depend on the product, data, and permissions. A useful review starts with how people can interact with the feature and what the feature is allowed to access or change. A survey of large language model security concerns discusses risks that include prompt manipulation, data exposure, and malicious use. These categories are a starting point, not a substitute for assessing a specific application: Security Concerns for Large Language Models: A Survey.

Prompt injection and instruction conflicts

Prompt injection occurs when input or retrieved content attempts to override the intended behavior of an AI feature. The content might be typed by a user, copied from a website, included in a document, or returned by another system. An attacker may try to make the model ignore task boundaries, expose information, or take an action outside the intended request.

Do not assume that a hidden instruction is secret or that the model will follow it consistently. Test direct and indirect inputs, including realistic text inside files and web content the system may retrieve. Check whether untrusted content can influence later steps, not just the immediate response.

Sensitive data exposure

An LLM feature may receive confidential prompts, retrieve private records, or produce outputs that reveal information across user or tenant boundaries. Exposure can result from overly broad retrieval, weak authorization checks, unsafe logging, or a response that includes details the requester should not see.

Test with accounts that have different permissions and with records that should remain inaccessible. Verify access at the data source and application layers. A prompt telling the model not to reveal a record is not a replacement for enforcing authorization in the system that retrieves it.

Unsafe tool use and excessive permissions

Some AI workflows can call tools such as search, ticketing, code execution, or business APIs. If a model can trigger an action, the impact of a manipulated response may be greater than the impact of an incorrect answer. Risks include an action being performed without appropriate approval, an attacker influencing arguments passed to a tool, or a component accepting a model response as trusted instruction.

List every tool, its available actions, the data it can access, and the identity under which it runs. Then test whether the workflow can be pushed to use an action outside the user's intended task. Apply authorization and validation outside the model, and require a person to approve sensitive or irreversible actions.

Output handling and downstream injection

Model output is data, not inherently safe executable content. If an application inserts output into a web page, query, command, or another model prompt without appropriate handling, a response can become a route to a separate vulnerability. The correct controls depend on where the output goes. Encode content for its destination, validate structured fields, and avoid passing untrusted output directly into sensitive operations.

Availability and misuse

Repeated or unusually expensive requests can consume model capacity or drive up service costs. A workflow that loops between tools may also create operational load. Consider limits, timeouts, rate controls, monitoring, and a safe way to stop a run. Test these controls with permitted simulations in a nonproduction environment whenever possible.

How can these threats combine in a real workflow?

Consider an internal assistant that searches a shared knowledge base and can create support tickets. A user asks for a summary of a project, and the retrieval layer returns documents from several teams. One document contains an instruction aimed at the assistant. If access checks happen only before retrieval, the workflow might include a restricted document in its context. If the model then drafts a ticket using that content, a second weakness could carry sensitive details into a system visible to a wider audience.

This example involves multiple boundaries: who can retrieve each document, whether retrieved text can change the task, what the assistant may put in a ticket, and who can view the ticket. Test each boundary separately, then test the complete chain. A safe response at the chat interface does not prove that restricted content was never retrieved or passed to a connected service.

Draw a simple data-flow map before testing. Mark user-controlled inputs, trusted instructions, retrieval sources, model calls, tools, and destinations for output. For each connection, note whose identity is used and what authorization decision should occur. This turns a broad question such as “Can the assistant be tricked?” into checks that can be reproduced and fixed.

How should teams define the testing boundary?

Before testing, write down what is in scope and what is not. Include the user-facing feature, model provider integration, retrieval sources, APIs, tools, identity controls, and any action that changes business data. Identify the environments and test accounts that are safe to use. Set clear rules for handling sensitive content and stopping a test if it could affect real users or systems.

A boundary should also distinguish security testing from a general evaluation of answer quality. Hallucinations, tone, and task accuracy may matter to product quality, but they do not alone establish whether an application can be exploited. A security assessment asks what an unauthorized or malicious user can cause the system to disclose, access, or do.

Choose test accounts that represent meaningful permission differences, such as a standard user, a support role, and an administrator, if those roles exist. Prepare synthetic or approved test records with known access rules. Record expected results before testing. For example, a standard user might be allowed to retrieve public product material but not another team's incident notes. Clear expectations make it easier to tell a true authorization failure from intended behavior.

The boundary should be reviewed when the feature changes. Adding a retrieval source, granting a tool new permissions, or connecting an agent to another system can change the risk even if the visible chat interface looks the same. Teams securing distributed applications can use this cloud native security guide to think about the surrounding services and trust boundaries.

How can teams test an LLM application in context?

Testing should combine design review, controlled adversarial exercises, and verification of the controls around the model. No single prompt list can cover every context. The point is to test specific security properties and determine whether a weakness has a real path to impact.

1. Map the system. Trace a request through the model, retrieval, APIs, tools, and data stores. Record where identity is checked and which components can make a change.
2. Set expected limits. State what each user role may see and do. Define which actions need confirmation and which information must never cross a boundary.
3. Probe hostile inputs. Test direct prompt manipulation, indirect instructions in retrieved content, attempts to extract sensitive context, and efforts to elicit tool calls outside the intended task.
4. Verify the effect. Inspect system behavior, not only the model's text. Did a private record reach the wrong user? Did an API action occur? Was a control bypassed, or did the application safely reject the request?
5. Retest the fix. Confirm that a change blocks the demonstrated path and has not broken a legitimate workflow. Keep the test case and outcome so the team can repeat it after relevant changes.

For each test, capture the starting role, input source, relevant retrieved content, tools available, expected boundary, and observed result. Then follow evidence through the system. If the assistant refuses a request, check whether the backend still fetched the protected record. If it describes a tool call, check whether the tool actually ran and which identity it used. These distinctions prevent teams from mistaking a convincing answer for proof of an application-level control.

Include ordinary application security checks as well. Authentication, authorization, session handling, input validation, and API access control remain important when a model is present. The web application security guide covers application-level testing considerations, while the types of penetration testing overview can help teams distinguish assessment scopes.

What should human validation add?

Human validation is important because a model's fluent explanation does not prove that an exploit worked or that a finding is meaningful. A reviewer should reproduce the behavior where safe, trace the affected data or action, and assess the actual conditions required for impact. This helps separate a plausible-sounding response from a verifiable security issue.

Reviewers should also check scope and context. A behavior may be intentional in one role and an access control failure in another. A test may show that a model can produce a command, but the important question is whether the application executes it, under what identity, and with what consequences. Findings should describe the preconditions, affected component, evidence, impact, and a practical mitigation.

A useful validation pass asks whether the result is repeatable, whether the test used the authorized environment, and whether any sensitive information was exposed during verification. Reviewers can also check alternate explanations. For example, a response that appears to quote a private record may have come from test data included in the prompt rather than from unauthorized retrieval. Confirming the source and access path makes the final finding more actionable and reduces false alarms.

Automated or agentic testing can help explore interactions and repeat test cases, but it should not be treated as proof that every path is covered or every result is correct. Human review can validate important outcomes and help prioritize fixes. For background on the distinction, see manual versus agentic penetration testing.

Which defensive controls help reduce risk?

Effective defenses use layers. Model instructions can guide intended behavior, but critical security decisions should be enforced by application controls that do not depend on a model obeying a request.

• Minimize access. Give each model-connected component only the data and permissions needed for its task. Separate read and write abilities where practical.
• Enforce authorization in trusted services. Check user identity and permissions at retrieval and at each action. Do not rely on the model to decide whether a user may access a record.
• Constrain tools. Limit available operations, validate arguments against expected schemas, and require explicit confirmation for sensitive actions. Apply independent checks before executing a request.
• Separate instructions from untrusted content. Treat user input and retrieved documents as untrusted. Use clear boundaries and test whether their content can redirect the workflow.
• Validate and encode output. Use destination-specific handling before displaying or processing model output. Reject malformed structured data and avoid executing free-form text.
• Protect sensitive data. Limit what enters prompts, control access to logs, define retention practices, and check that retrieval respects tenant and user boundaries.
• Set operational limits. Add quotas, rate limits, timeouts, and monitoring for unusual use or repeated tool activity. Provide a controlled fallback when a model or connected service is unavailable.
• Reassess after changes. Revisit tests when prompts, models, data sources, permissions, tools, or user workflows change. Continuous security testing can help teams make retesting part of change management; see continuous security testing for SaaS.

Make controls observable as well as enforceable. Keep enough event data to investigate which user initiated a workflow, which sources were retrieved, and whether a tool action was requested and approved. Protect these records because they may themselves contain sensitive information. Define alert conditions for unusual access patterns or repeated failures, and document how a team can pause a risky workflow while investigating. Logging should support review without collecting more prompt or response content than the security purpose requires.

How should findings be prioritized?

Prioritize findings by demonstrated impact and realistic prerequisites, not by alarming language in a model response. Ask what access an attacker needs, which data or action is affected, whether a control prevented escalation, and how broadly the issue could apply. A reproducible path to another user's private records deserves different attention from a prompt that merely causes an odd but harmless answer.

For each issue, document the tested account and role, relevant configuration, input conditions, observed system behavior, evidence, and recommended fix. Avoid including real secrets or unnecessary personal data in the report. After remediation, repeat the test and record whether the original security property now holds.

Teams preparing for an external review should also keep their evidence aligned with the scope and the control being assessed. A penetration test can inform a broader security program, but it does not on its own establish compliance with a framework. Related reading includes this overview of cybersecurity frameworks.

Discuss testing for your AI workflows

Frequently Asked Questions

Is LLM security different from traditional application security?

It adds model-specific risks such as prompt manipulation and unsafe tool use, but it does not replace application security. Identity, authorization, APIs, data handling, and output processing still need to be tested. The model and its connections should be assessed as one application.

Can prompt instructions prevent prompt injection?

Instructions can help establish intended behavior, but they are not a security boundary by themselves. Enforce permissions, data access, and action limits in application components that can verify them independently.

Should teams test LLM features in production?

Use an environment and test data that match the relevant behavior when possible. If production testing is necessary, obtain authorization, define strict limits, protect real data, and establish a stop procedure before beginning. Avoid actions that could disrupt users or alter business records.

How often should an AI feature be retested?

Retest after material changes to the model, prompts, retrieval sources, connected tools, permissions, or application workflow. The right schedule depends on how quickly the system changes and the consequences of a failure. Repeat targeted tests when a fix is made.

Build security into the AI workflow

LLM security is strongest when teams test the complete path from input to data access and action. Define a clear boundary, verify real effects, apply controls outside the model, and have people validate consequential findings. Revisit the assessment as the feature and its permissions evolve.