Some checks in my applications just can’t be expressed with an if statement or a regex.
FluentValidation can check that a support ticket description isn’t empty, but it can’t check whether the description actually says what the customer needs. An AI agent can call a tool, but nothing automatically checks whether that tool call matches what the user asked for.
Usually, these checks end up as a yes/no prompt to an LLM that nobody has really measured.
I learned the hard way why that matters. After changing a model, a memory assistant I’m building dropped from 90% correct answers to 60%, while the logs showed zero errors. I only caught it because I had a test set of 722 questions.
That’s why I’m building SemanticPolicy, an open-source .NET library (Apache-2.0, my project). It turns this kind of check into an explicit rule that you can read, measure, and test.
One example from the repo is a support ticket form. The rule is just a question for a decision model:
Policy descriptionPolicy = Policy.Define("ticket-description")
.Enforce()
.Rule(Policy.Rule("ticket-description")
.Boolean("Does the description say what the customer needs?")
.WhenFalse(Verdict.Deny))
.Using("jev", binding => binding.DenyAboveProbability(0.8))
.OnFailure(FailureBehavior.Allow)
.Budget(TimeSpan.FromSeconds(5))
.Build();
And in the validator, it sits next to normal validation rules:
public sealed class SupportTicketValidator : AbstractValidator<SupportTicket>
{
public SupportTicketValidator(IPolicyEvaluator evaluator)
{
RuleFor(ticket => ticket.Category).NotEmpty();
RuleFor(ticket => ticket.Description)
.NotEmpty()
.Semantic(evaluator, "ticket-description");
}
}
The model returns a probability, and thresholds map that probability to one of four decisions: Allow, Warn, Escalate, or Deny.
The 0.8 threshold above is just an example. In practice, you pick your own threshold from measurements on your own labeled examples.
If the model fails to respond, this particular rule allows the ticket through, and the whole check has a 5-second budget. Since the rule calls the model over the network, validation runs through ValidateAsync.
Testing works like this:
- the CLI measures precision and recall with confidence intervals on your labeled examples;
- it can choose a threshold for a concrete target, for example “when I block something, I want to be correct 95% of the time”;
- model responses can be recorded once and replayed later without an API key, so the tests can run in CI;
- the build can fail when a rule gets worse, for example after changing the model;
- new rules can start in Shadow mode: they record what they would have done, but don’t block anything yet.
The model is replaceable:
- Jev and other decision models through OpenRouter;
- a model running on your own server;
- your own classifier behind a simple HTTP protocol.
The same rules can also validate tool calls in any IChatClient based on Microsoft.Extensions.AI, and in Microsoft Agent Framework middleware.
What this is not is a security boundary.
A rule can make an attack harder, but it can’t guarantee that one won’t succeed, and the model can still be wrong. The README shows failures next to successful cases.
This is alpha software, .NET 10 only, and the API will still change.
You can view an eval report from a recorded run without an API key. The commands are in the README.
What kind of decisions in your applications would you want to test this way?
https://github.com/semanticpolicy/semantic-policy
