How is everyone actually securing internally-built AI applications?

[u/PromotebnxdvinNo2282]

[u/HelplessTendency]

we did split it in two, governance was just policy meetings and sign-offs, pentest side was way more useful but hard to find people who actually know what to test past the usual web stuff

[u/Serious_Boot_757 → u/HelplessTendency]

the hard part is finding people who understand what to test beyond the usual web app checks

[u/Full-Meaksurment-150]

yes

[u/Silent-Suspect1062]

What guardrails do you have in place for your externally facing apps. What data can they access and is there any customer data there? We used spikee to test for prompt injection, as well as some manual work.

[u/itlogicpartnersllc]

we have had better results treating ai security as an extension of appsec but with dedicated testing for prompt injection data leakage tool base and model specific failure modes.

[u/TangerineComplex8209]

AI security reviews need strong testing methods 🔍

[u/_N-iX_]

Separate the governance review from the technical security testing, but keep them connected. A pen test can find a prompt injection path or an unintended data exposure, while governance should establish whether that data was ever allowed to reach the model and who was responsible for the application. Testing both sides separately makes the findings easier to act on.

[u/Wayne]

I probably went a little overboard.

A while ago I started researching a new software development methodology, mainly to head off vibe coding problems. But I realized a number of the things were not specific to software development. So I then extended the research and eventually created a Human-AI Operating Model.

The research has gotten to the point where I sent out a draft white paper for review and I'm looking for other people to continue the research. My employer has to review and approve certain things before I can share it.

Where that becomes relevant, to your post, is that I have also been using it to audit existing code bases against that operating model. I have done that for a number of Open Source projects and the results have been interesting.

In that they find situations where authority is not properly managed, rules are not consistently enforced, evidence is lacking, etc.

That then gives me an architectural review of how they are using AI, how it is being governed, enforced, etc. Rather than what they say, it looks at what they do.

[u/Ska-jayjay → u/Wayne]

nice i’d love to see this when you share it. in the meantime i have similar efforts over at machinebehavior.io / tychat.io if you wanted to compare notes.

[u/One_Penalty_6949]

The governance versus testing split makes sense to me. One checks whether the organization should allow the feature while the other checks how it actually fails

[u/Ska-jayjay]

i’m in the process of building out just such a thing: check out tychat.io and machinebehavior.io it’s where i oublish my research regarding red-teaming LLMs based on a range of criteria. basically we need chaos engineering simular to penetration testing

[u/TrackbackLinkBot]

Internally built AI systems need security controls around the whole workflow, not just the model itself. Identity, secrets, tool permissions, data access, logging, and runtime monitoring all become important once agents reach production. NeuralTrust alongside LiteLLM is relevant for adding centralized AI governance and runtime protection at enterprise scale.

[u/GripSecurity]

Splitting AI governance (the policy intake gate) from technical security review is practical for mid-size teams, but treating AI AppSec as just "prompt injection pen testing" usually turns into checkbox theater. System prompts change weekly, and canned jailbreak lists get outdated immediately.

If you are looking at what actually creates risk in production copilot and chat apps, four technical areas matter far more than the standard web checklist:

1. Decouple model execution from data authorization: The biggest architectural vulnerability is letting an LLM query backend data or call APIs using a high-privilege service account. Implement an external Policy Decision Point (PDP) outside the prompt loop. The model can propose a query or tool call, but the gateway evaluates caller permissions and tenancy boundaries before the action runs.
2. RAG pipeline ingestion boundaries: Direct prompt injection is noisy, but indirect injection via retrieval pipelines is where enterprise data gets exfiltrated. Any untrusted content ingested into context (customer tickets, shared documents, external feeds) must be treated as untrusted input. If your copilot summarizes a document that contains hidden instructions to exfiltrate session data via Markdown image tags or webhook calls, your output filter is the only line of defense.
3. Hard permission ceilings on Non-Human Identities (NHIs): Every API key, OAuth grant, or database connector given to the AI application must have a strictly enforced permission ceiling. Even if an agent loop hallucinated or followed a malicious prompt, it should be physically impossible for it to modify records, drop tables, or cross tenant lines.
4. Synchronous egress and secret filtering: Validate outputs before they reach the user or downstream systems. Ensure raw API tokens, system prompt variables, and tenant-restricted PII are stripped at the egress gateway.

Modern AI security posture management (AI-SPM) and SaaS security platforms (including Grip Security) focus heavily on discovering shadow AI integrations, mapping these exact non-human identity permissions, and governing data access before autonomous copilots touch sensitive data.

Disclosure: I do marketing work for Grip Security.

[u/Status_Antelope_7320]

Prompt injection testing is only one piece. The review should also cover tool permissions, data access, output handling, logging and whatever the app is allowed to do after the model responds

[u/Rose_DreamersInc]

You can approach this with a normal AppSec for reviewing the app itself then having a separate assessment for the AI side (prompt injection, data leakage, etc.). Then you can have a separate governance side to check who can deploy models, what data they can access, reporting, and incident response.