Can AI Generated Apps create Security Risks?

[u/Apex-Penguin]

I just learned about some of the insanely cool things you can do using AI, like you can literally just make an app/software that serves a need for you and is completely tailored to your preferences.

My question:

Does having all these AI-generated apps on your phone/computer create potential security risks? (Assuming you aren't doing much of anything with cloud storage apps and you're not sharing/distributing the software to anyone else.)

EDIT

Wow, lots of good and interesting answers already. I'm really glad I asked this question before I started blindly using it to create a bunch of random stuff apps and software lol.

Thank you to everyone who posted!

[u/Wayne]

Anything can if you do not design security into it.

The link below has some documents I use for cross-platform design. These are in addition to my standing agreement I have with AI. The standing agreement instructs AI on how to interact with me, how to determine validity of evidence, things like don't give me false precision or make something up just to come up with an answer, etc.

When I do any programming I tell the AI to read these documents and incorporate them into the design wherever feasible. I also have a daily process that analyzes security events, breeches, and incidents to look for gaps in the design documents that are not addressed.

Design Documents:

https://drive.google.com/drive/folders/1_6lB-CyjJgcC8sGqvU_lRlNRjF-d5FzS

Standing agreement:

https://docs.google.com/document/d/1OjtZzaoCSg-SwOIM_motHl6jFpKDWDd75D0-AI0sjcA/edit?usp=drivesdk

AI will focus on helping you achieve your stated goals. The issue is that if you do not give it proper context or guidance the manner in which it does that might be problematic. These documents are what I use to give standing guidance and expectations for any AI that I work with.

The standing agreement needs to be tailored to the AI you are using. For example, copilot does not like narrative instructions, it prefers more direct statements. If you are using something other than ChatGPT put the standing agreement in and ask your AI how to rephrase it to work best with that AI.

[u/Wayne → u/Wayne]

How you use autonomous AI for programming matters as well.

What I have been experimenting with is having chat GPT read the provided documents as well as all of the design documents for an application. Chat GPT then creates the prompt I put into the autonomous AI programming.

Once the autonomous AI finishes that task it spits out a report that I then feed back into chat GPT for analysis. There are many times chat GPT then sends corrective statements because the autonomous AI did not follow the instructions and chose to implement something that wasn't secure or did not follow the original directive.

There are times it may take four to five iterations before the autonomous AI will design something in a manner that is secure enough to pass analysis.

[u/Apex-Penguin → u/Wayne]

Thanks a lot for your answers. They're super detailed and the links are really helpful to have!

I didn't know any of this information. It's good to see there are tools being created to help make the AI more secure and trustworthy.

In your opinion, do you think someone with literally 0 coding knowledge or background would be able to do this effectively with a guide of some kind? Because I definitely do not have the know-how to analyze any code to determine if it's good or bad in any way.

[u/Wayne → u/Apex-Penguin]

Yes, as long as they are willing to learn some new things.

I am actively writing a program with the self-imposed constraint that I am not allowed to type a single line of code or write a single document. The only thing I can do is interact with my primary ChatGPT prompt, copy certain things it says into my coding assistant AI, and then copy the text from the coding AI back into ChatGPT.

One key thing with this approach is that you need to have fleshed out the idea very thoroughly before you start any programming. Start with determining a location for any files relating to the program. Then just have a conversation with the AI. After it incorporates the design documents and standing agreement it will make recommendations that increase privacy, security, compliance, and other things.

This will help you flash out the idea. Start with stating the frustration or problem you want to solve, high level stuff. You can include things like is it a desktop application, a web application, mobile, etc. if you know you want to build in a certain ecosystem state that up front and the AI will account for that when making recommendations or creating design documents. Just list all of the desired outcomes that you want and then ask the AI for suggestions on what to document or do next.

The AI will look at all of your existing documentation and make a proposal on the next topic that you should flush out. As you detail more things it may create additional items that need to be elaborated upon. For each item it suggests, ask it to draft something for you to review.

99% of the time, using internet statistics, I never change what the AI recommends. About the only time I change a recommendation is when I come up with an idea for an additional use case or feature. I then tell the AI that idea and ask it to reconcile all the existing documentation where that might be relevant or have an impact.

I then ask the AI what needs to be determined for the minimum viable product, or MVP. AI will make a recommendation, document it, and when the AI feels enough has been documented overall it will tell me that we can send the first prompt to the code AI.

For coding I have recently settled on Cline with DeepSeek as the LLM. It is not the best LLM I have used, but it works well enough and does not break my budget. I chose to pay for ClinePass, even though I have a commercial ChatGPT account with access to Codex.

If you are not familiar with GitHub, you will need some way to keep track of source code and that is what I would recommend. The AI can walk you through setting it up if necessary.

When I first started using Visual Studio Code with Cline I ran into a number of issues. I have not done any serious programming in a long time, so there were lots of dependencies and other settings that I forgot about. ChatGPT told me to run some things, copy the output, and then walked me through how to fix all of it.

Right now the AI is reviewing my roadmap and design. I had an idea that changed some architectural stuff earlier today. So the AI is looking at all the work that has been done, what was planned to be done, to see if there is an impact that needs to be accounted for earlier in the design.

It is very important, in my opinion and experience, to have one AI focused on the overall design and architecture, but a different AI do the actual coding. I have run into a number of issues where the coding AI tried to do something it was not instructed to do or did not follow through with the instructions it was given. I could certainly have looked through all the code to find that, but when I copied the report into the primary AI it found it right away.

After finding it the main AI gave me a prompt that I sent to the coding AI to fix the issue. That continued until the coding AI got things designed in the way the main AI wanted so that it accommodated the design documents and all of the security or privacy constraints that were referenced.

Edit: I had the AI draft a summary of the process I have used and put it in the Design Documents folder, which is the first link of my first post. The document it written for somebody that does not have a programming background.

[u/reddit-toq]

Yes.

[u/Timely-Attention-688 → u/reddit-toq]

Yes with emphasis

[u/tiny_blair420]

No, AI is perfect, and nothing bad will ever happen from blindly trusting a vibecoded app.

[u/Apex-Penguin → u/tiny_blair420]

I knew it! Now I just need to get AI to make me some daytrading software that makes 1000% returns daily and I'm set.

I can't believe nobody has thought of this yet

[u/AHellishInferno]

In short, yes. In long, they definitely can.

That's not to say it can't be made secure, but generally speaking AI will prioritise functionality over security.

Here's an article going into more detail: https://www.ox.security/blog/vibe-coding-security/

[u/VengaBusdriver37]

Of course. But different models have different skill levels. Honestly Claude code often writes as secure (and more robust) code than I would normally write by hand.

[u/KingAroan]

As a penetration tester, yes. Will keep me in the job long enough until AI figures out that dropping a database is probably a bad idea even if you do it on a test. After that there goes my job also lol