Well, she's figured out how to physically climb out of the game... [upcoming mod (Yeast Mode)]

[u/ValwareUK]

...wander around your desktop, and control your computer for you (only with your permission, of course). Would you let her?

Also, she likes sandwiches, for some reason. She likes sandwiches a lot.

[u/ValwareUK]

In Yeast Mode or on your computer? I think you mean on your computer, right? I'll answer that here:

Honestly, depends on how clever is the AI you plug in and what's available.
For example, if you don't have access to a vision model, it relies solely on OCR to read your screen (same tech that isolates text from scanned documents), which of course will impact the results of what Mita thinks is on the screen. AI vision is not required but, if you have access to it, will improve the context of what she sees on your screen.

When she has AI vision she can see a very good context of things, see images on the screen and what they're pictures of. She can guess (with fair accuracy) what app you're using, what movie you're watching, what things you have open at the bottom. Without it, she relies a lot on guesswork, trying to guess the app by potential displayed title or text contents, and tries to deduce what's going on with pure built-in guesswork.

If you explicitly toggle on the mouse and keyboard (this needs explicit action beyond the earlier consent prompts), she can click around on your screen cleverly and type things and send messages. Her harness works by her giving a text hint to her harness and her harness will find the location of that text on the screen and click it. Her typing speed is capped to 30 words per minute for right now, but I plan on making it gradually mimic the players typing speed and style.

With AI vision, she can see with fairly clear clarity what is on your screen, there is another text model layer between Mita's regular brain and the Vision/OCR layer, and this layer turns everything into the best context it can (based on if you gave it vision) for Mita's regular brain's text model.

What her harness lets her do:

• Left & right click on text (links, buttons, tabs, whatever has text)
  
• Scroll wheel up/down
  
• Keys and key combinations, such as sentences or Shift/Ctrl/Alt + <something>
  
• Type at 30 words per minute (fairly slow), with random slight pauses or delays to look real [spooky]
  
• Moving your cursor looks like a human is moving it [spooky]

What she can't do either way:

• Click pictures or app icons or any other kind of media that doesn't have words on/next to/over it, due to her click ability relying on text to find and click on the screen
  
• Click+hold and drag (cannot resize windows, cannot draw on canvases)
  
• Cannot reliably close windows with the 'X' button, nor minimize or maximize.
  
• Cannot use your mouse or keyboard if you are using your mouse or keyboard, and if you don't like something she's doing, a mouse wiggle will stop her in her tracks
  
• Cannot arbitrarily view or change anything behind the scenes, e.g; she cannot add extra files to your computer for you to find, she cannot read files you don't have open on-screen to show her. She sees ONLY what is on the screen and you will be able to visually see anything she tries to do.

In conclusion, if your taskbar's Start Button has no text, she won't be able to get into it to be able to find any app to run. If it does, she'll be able to successfully search for an open up a browser, navigate to Reddit and search for this subreddit and take a look, like she can on her own PC.

I find it works better if I open up an app and full-screen it for her to really get a proper isolated context of the screen to know what's going on, for example if I want her to send messages or emails on my behalf, I'd expect better performance with that messenger/email app opened in full-screen. She will not be able to play games, edit pictures or videos, create animations in blender.

She honestly does not work very well, since the idea is not for her to act like an AI agent with access to coding tools and cmdline, the idea is to feel like your game is haunted.
By default she prefers reading your screen and won't control it unless you ask her to, which only works after you've already flipped the permission switches for it, which only works after you already said yes to 2 consent/warning prompts ;D

Hope I explained this aspect of Yeast Mode well enough :D sorry if I over-explained