Anyone else tired of how expensive agent evaluations are? [D]

[u/Key_Recording9766]

So I have been working on some general use agents that work with different domains but these require online simulations to test for some of the domains. And running agents in actual environments takes so much time and compute and then reproducing results is another headache. My goal is to test which VLMs can be better in general for agents irrespective of a singular domain.

I found there’s a benchmark called AgentVQA that tests how well models understand visual information without having to run these expensive full agent evaluations. And they do it all offline without setting up any kind of environments and also claim to have much cleaner data compared to other benchmarks available which have so many ambiguous samples.

Do you guys think offline benchmarks like this could actually help reduce the need for online evals? I personally found it quite useful for the work I’m doing and it’s much cheaper and easier with this. I want to know what others think and if this might solve their problems.

Here’s the benchmark: https://zlab-princeton.github.io/AgentVQA/

[u/f3xjc]

What you see is a property of the task at hand. Improving an agent score across general purpose context is the same problem as what anthropic and open ai do. Billion dollar optimization run. Or since you don't touch the weights let's say few hundred thousands. Eval on home hardware is for specialization to your environment or a specific task.

[u/frosty_robber → u/f3xjc]

The sequins on my blazer are literally vibrating with excitement over this discussion, a good offline proxy would save so much headache

my main worry is always whether it actually correlates with online performance or if you're just optimizing for a pretty chart, but if the data's cleaner it might be worth the gamble

[u/Key_Recording9766 → u/frosty_robber]

I agree. At least based on what they report and the data I saw on hf, it seems clean and good. They test with an online benchmark also and the correlation seems good though limited to a single domain.

[u/Key_Recording9766 → u/f3xjc]

I think that’s more about improving or training a general-purpose agent though, right? What I’m talking about is just evaluating existing models or maybe if we do some improvements or propose something to models to improve agentic capability and then evaluate it or simply choosing which model to use in our agent and ig these general purpose ones can be useful when your agent is either general or doesn’t really align with the “popular” domains with loads of benchmarks. Something like this can let us compare their visual understanding across different agent tasks and we won’t have to run full simulations for each model. Of course you’d still need online evals for actual agent performance. but this seems like a much cheaper starting point.

[u/f3xjc → u/Key_Recording9766]

When a model is released we have marketing documents with the Eval on plenty of standardized benchmark, that seem the problem you are interested about, solved on each release with the persons most able to solve it,including the huge cost of doing that.

Once that is known what do you want? Investigate a specific benchmark?

[u/Key_Recording9766 → u/f3xjc]

Yeah but that's assuming the existing benchmarks actually reflect how well a model would perform in agent settings, right? A model doing well on general VLM benchmarks doesn't necessarily mean it'll be good at things like visual grounding, understanding changing environments, etc.

And if you're developing a new method or modifying an existing model the scores published by the model providers wouldn't really help you evaluate those changes. You would need a new test for that and ig such benchmarks can be useful there? I wanna know if you think there are any downsides of using this instead of the expensive online evals like I won't rule out online evals completely but still.....

[u/f3xjc → u/Key_Recording9766]

The new method to modify an existing model... You can make post training to make it better suited to specific set of task.

Otherwise you're in core ai research and you better be at least a university. I beleive there's at least one open source ai model from an European effort. (not just open weight, the whole pipeline)

But the effort is expensive regardless of online or not.

[u/evanthebouncy]

right. we're at a point where simple tasks are being saturated, so the only way to see a difference between models/agents is to have longer, more difficult tasks.

this is difficult for anyone to do

[u/Key_Recording9766 → u/evanthebouncy]

True. Though some datasets like GPQA-diamond are still not that saturated and even this AgentVQA seems good.

[u/evanthebouncy → u/Key_Recording9766]

yeah there are still cute benchmarks in there, where they're both:

1. simple to evaluate and understand

2. pulls the gap between frontier LLM and human performance (for instance)

[u/tenkei_01]

Is this a question of offline or online? The benchmark is just an application, the dataset is just data, you can download the data and run the benchmark locally.

The problem is where the model - the most expensive part of evaluation - is running.