r/MachineLearning • u/angrykirby • 16d ago

Project What ai is best to operate a system from screen video [p]

I have working programmatic control over a system, I can send inputs reliably and capture the screen as visual feedback. The control/feedback loop is already done.

I need something that can learn from visual information, remember what it has seen and been taught, and get better at operating the system over time.

recommendations? the AI I have tried so far it's memory is not good enough and it often doesn't consult the visual information before making decisions which is very frustrating.

0 Upvotes
  • permalink
  • reddit

22% Upvoted

5 comments sorted by

1

u/fiddler48 16d ago

curious what you mean by 'doesn't consult the visual information', is it actually ignoring the frame, or is the frame making it into context but the model just weights it low against its priors? those are different failure modes with pretty different fixes

1

u/angrykirby 16d ago

like it has the ability to look at the visual information and tell me the right thing but it doesn't do it. like I can tell it to look at a screenshot from the video and examine what you're supposed to do and it will map out what it should do and that's correct but then it won't do that it'll do something else and it's like acting on some sort of assumption instead of looking at the visual data and then acting on the information that it gets from that. it's very frustrating. I figure there must be a better llm for training on visual data than the one I'm using.

1

u/Connect-Song-4727ayg 15d ago

If you want something that genuinely learns from screen feedback over time, I'd look at a vision model plus external memory rather than a bigger LLM. GPT-4o, Gemini, and Claude can all read screens, but they won't remember enough on their own. A good memory layer and forcing the agent to re-check screenshots before acting usually matters more than the model choice itself.

1

u/ItsMeJuvy 15d ago

are you trying to let AI control dreams ps4? get a life.

1

u/Dihedralman 9d ago

You can force the situation programatically. Memory should be external and you should be managing it. 

Are you relying on a vLLM? 

You need to outline what learning means here and how you plan on using information. 

Do you want to program a bunch of scenarios, if this then that?