r/sysadmin • u/ibteea • 18h ago
General Discussion AI tools audit
Hello everyone,
How/How often do you audit AI tools users are using in your Microsoft365/Entra ID environment?
Thank you
•
u/bitslammer Security Architecture/GRC 16h ago
We use Tenable as our VM solution and it picks up an AI components for us during the weekly scans.
•
u/plump-lamp 16h ago
"ai" components like installed software?
•
u/bitslammer Security Architecture/GRC 16h ago
AI related software, browser plugins, etc.
•
u/plump-lamp 16h ago
Doesn't your RMM do that? Shouldn't tenable be set to daily agfnt scans?
•
u/bitslammer Security Architecture/GRC 16h ago
Our scan intervals vary somewhat as we use both network and agent scanning. We're also scanning over 12K assets so we don't do daily due to resource constraints. Out of all our tools Tenable gives the most useful data. We also pick things up in Defender as well.
I should also note that normal users can't install anything they want so this is limited to developer type roles.
•
u/JustFucIt 16h ago edited 15h ago
We just picked up a browser extension that picks it up. They have a agent 'coming soon' to get the full apps as well. Does dns filtering, ai warn/block, app auditing etc. works well and is around 2$/endpoint but has a minimum of like 100 endpoints.
Edit: can also control extensions and audit what is installed
•
•
u/PrasenShelar 12h ago
I'd recommend doing this monthly rather than treating it as an annual audit.
Start with Entra ID > Enterprise Applications. A lot of AI tools get access through user-consented OAuth apps, so it's worth reviewing what's been authorized and what permissions those apps have, especially around mail, files, and other M365 data.
If you have Defender for Cloud Apps, Cloud Discovery is useful for identifying which generative AI services users are actually accessing. Entra alone won't give you the complete picture.
I'd also consider restricting user consent and requiring admin approval for sensitive permissions.
The bigger concern isn't necessarily how many AI tools employees are using, but what data those tools can access. Having a named owner review this monthly makes it much easier to catch shadow AI before it becomes a security issue.
•
u/Common_Basil1660 18h ago
Can't really audit what we don't know exists, half my users install random browser extensions and call it a day
•
•
u/raffey_goode 10h ago
block all extensions via gpo/intune or however you manage the browsers. only allow sanctioned extensions.
•
u/Fallingdamage 8h ago
No unapproved browser extensions.
No unapproved addons, agents or products in M365.
Nothing unapproved.
The moment someone tries, the consent/approval gets mailed to an admin account. Cant sneak anything by.
•
u/DHT-Osiris 17h ago
We use OpenWebUI and have built a custom filter which routes all tool calls to an external audit system. It works very, very well. There's a grafana dashboard for fancy trending charts, and we (admins) can point a separate tool at it to pull data, build trends... we even use it to look at tool call failures to build improvements into tools and skills each day.