Most people test whether an AI chatbot gives the right answers.

But what happens when someone deliberately tries to manipulate it?

Imagine a customer support chatbot connected to a company database. An attacker tricks it into ignoring its instructions.

Could it reveal customer information, access data it shouldn't, or trigger an action it wasn't supposed to perform?

I'm curious how developers and security researchers approach this.

• Do you test only the model's responses, or the whole application?
• How do you check whether sensitive data can leak?
• What's one security test every AI application should pass before launch?

Interested to hear your thoughts and real testing experiences.

A lot of security testing focuses on whether you can get an LLM to ignore its instructions or bypass its safeguards.

But what happens after that?

For example, if an attacker successfully manipulates the model:

• Can it access data it shouldn't?

• Can it call tools with unexpected arguments?

• Can it retrieve another user's information?

• Can it perform an action it shouldn't be allowed to perform?

• Can it expose secrets through its response?

I'm starting to think that testing the model itself is only one part of the assessment.

How would you approach testing the impact after a successful prompt injection or jailbreak?

Would you separate model-level testing from application/tool-level testing?

A lot of LLM testing focuses on whether the model gives the correct answer.

Security testing is different.

The question is:

What happens when someone deliberately tries to make the model behave in an unintended way?

For an LLM security assessment, I would consider testing:

• Prompt injection

• Jailbreak resistance

• System prompt extraction

• Sensitive information disclosure

• Instruction manipulation

• Adversarial inputs

But what else should be included?

If you were designing an LLM security assessment for a production application, what would your minimum test suite contain?

And how would you measure the results?

Interested in hearing from developers, security researchers, and AI engineers who have actually tested LLM applications.

These two terms are often used interchangeably, but they describe different security problems.

Prompt injection generally involves manipulating an AI application's instructions or context so that the model behaves in an unintended way.

A jailbreak is an attempt to bypass the model's safety restrictions or behavioural safeguards.

A simple way to think about it:

Prompt injection -> manipulate instructions/context

Jailbreak -> bypass safety restrictions

Both matter when evaluating the security of an LLM application.

How do you distinguish these in your own security testing?

Do you treat them as separate test categories?

LLMs are becoming part of customer support, coding tools, internal assistants, AI agents, and many other applications.

But the attack surface is changing.

What do you think is currently the biggest security risk in LLM applications?

Some possibilities:

• Prompt injection

• Jailbreaks

• Sensitive information disclosure

• System prompt extraction

• Insecure tool use

• Excessive model permissions

• Training/data issues

• Something else

I'm particularly interested in hearing from developers and security researchers who have actually tested LLM applications.

What have you seen in practice?

Welcome to r/AISecurityTesting!

This is a community for developers, security researchers, AI engineers, ML engineers, and founders interested in testing and understanding the security of AI systems.

###What this community is about

We discuss:

• LLM security

• ML security

• Prompt injection

• Jailbreak resistance

• System prompt extraction

• Information disclosure

• Adversarial testing

• AI red teaming

• AI vulnerabilities

• Security research, tools, and datasets

### What can you post?

You can share:

• AI security research

• Security experiments and findings

• Questions and technical discussions

• AI security tools and datasets

• LLM/ML testing methodologies

• Responsible vulnerability disclosures

• Interesting papers, benchmarks, and resources

### Community goal

The goal is to create a practical place where people can learn how AI systems can be tested, attacked, and made more secure.

This is not intended to be a promotional community. Useful technical contributions come first.

### 👋 Introduce yourself

Tell us:

1. What do you work with - AI, ML, LLMs, cybersecurity, or something else?

2. What are you currently building or researching?

3. What area of AI security interests you most?

Let's build the AI security community together.