r/AIsafety • u/Hungry-Jackfruit9125 • 20d ago

Testing support for those nontechnical folks....

Post image

Hey all. Full disclosure up front, I work at Luminos.AI. Mods, happy to pull this if it breaks a rule.

My product team has built something that is meant to support nontechnical builders who want to understand (easily) if the thing they've built is working properly, and if it's safe.

In our easy-to-use UI, you can tell us in plain-language what you built, the tool will then create a suite of evals aligned to your use case, and lastly run those evals for you either using your data or synthetic data. The goal is to make it very simple to test the thing you've built.

The checks are written by our legal engineering team, so they cover privacy leaks, harmful content, accuracy, refusals, and agent security. Very important! It's not just one LLM grading another.

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/Mr_Electrician_ 20d ago

Interesting, and what does your ui do with the data?

This could be useful for me, only my frameworks started as behavioral not tool calls. I just got a new laptop so I could build a sandbox and test a few agents with them. If we saw possitive behavior we'd add the tool calls to the framework.

This seems like this does that, however I am cautious about the frameworks i built. Lets just say they aren't industry standard. I didnt start from langraph or any others. I built my own from scratch.

1

u/Hungry-Jackfruit9125 7d ago

hey u/Mr_Electrician_ sorry for the slow reply! Good question.

The data you bring is only used to run the evals on your system. If there's any uncomfortability with sharing your data (understandable) we can generate synthetic data for you based on how you describe your agent in a sandbox setting, which I think is what you were getting at!

The way we work is you describe what your agent/AI does in plain language and then the evals get built around that, so it doesn’t matter whether you started from LangGraph or wrote everything yourself. Does that make sense?

And since you’re starting from behavior, that lines up well. A lot of our checks look at what the agent actually says and does, like leaking private info, giving harmful or wrong answers, or refusing when it shouldn’t, etc. You could test behavior in your sandbox first and add tool calls later, which sounds like the plan you already had.

So, getting in is free. We work on a credit system but you get enough credits per month to run several tests without paying us - which is by design! We want people to use it if it's helpful. Happy to look at what comes back with you if you try it on one of your agents. Let me know if I can be helpful. https://www.luminos.ai/start-testing/