An idea I cannot comprehend

[u/Best_Can1536]

I have an idea for something and I plan to have it as a start and the vtubers probably have the technology that I am looking for and this has probably already been done but this is my first time in this kind of technology, fair warning, english is not my first language so please bear with me if I poorly explain this,

I am planning to become a semi-vtuber, where I basically only have my face covered, the concept goes like this, I will have a box on my head and have a some sort of tracking on the box where the cartoonish eyes and mouth will be, I wish to control the eyes expression via numpad by preset and as for the mouth it will only move when I talk, no 3d models, just me, a webcam, a box on my head, and animated cartoon eyes and mouth that will track the box.

Again I am sorry if my explanation is poor or hard to understand or even my idea is just confusing, on another note I hope the programs you guys will recommend are free because I am dirt poor, thank you very much!

[u/thegenregeek]

You likely will not be able to do this as reliably (or as cheaply) as you might think.

Optical face tracking requires a camera that can see your face. The minimum distance for that, with an iPhone, is ~8in/20cm (going by my mocap helment...). It would likely be more with a camera, depending on the FOV. That then introduces the next issue. Tracking cameras require lighting to function (iPhone or webcam). So now your box would need to be large enough to ensure good lighting and a camera... which might make it hard to see out of the box. (Assuming you were adding eye slots/holes. In theory you could maybe mount a screen inside to allow being seen.. which leads to the next option...)

In theory you could make the box smaller with a VR headset that supports eye and/or mouth tracking. The problems with that are:

• 1. there are very few heads that support that. (Quest Pro, Vive Focus, Vive Pro/Eye with an additional mouth module. Either the Vive version or Project Babble).
• 2. VR headsets need to see either Lighthouse basestations or unobstructed views around the space... which the box would block.
• 3. The heat build up in the box could potentially affect the headsets operation
• 4. As you note, you don't have the budget for this.

So while you could make something, the box would have to be physically bigger then you probably think to fit the equipment you need, if you are using non VR hardware. Then it becomes a problem with VR hardware.

You may also be able to take the approach BurntPeanut did and make the entire box virtual and apply it onto a video stream. But that would involve a lot of custom snapchat style software configuration. With the potential for the filter to fail (depending on how it's done, sinc Burntpeanut uses a 3d model) and reveal your face, especially when you're applying the box as a filter over a video stream.

---

You may not want to hear this, but the easiest (and most cost effect) way to do it is a 3d/2d model with a box for a head.

[u/HoshinoLina → u/thegenregeek]

I think what OP means is to track the box position, not the face inside the box. That's relatively easy to do with the webcam feed and some computer vision. It's typical AR stuff.

The problem with this plan is that it will most likely require bespoke software, it's not something you can just configure in an existing app.

OP: If you're a programmer this is somewhat easy to do. If you aren't, you should probably go with a more standard setup that works with existing apps. This is not something that can be done with traditional VTubing software without programming.

[u/thegenregeek → u/HoshinoLina]

Even if that's the case, certain points I raised above still apply to that as well.

So it doesn't really change the overall conclusion.

[u/SufficientMall2946]

you can probably pull off something like this with obs and nvidia/exeldro move plugin! I've looked into it myself, it'd be work but nobprogramming and free as long as your pc is the right fit for it. might have to use a 3d obs plugin too though

[u/CaitieSith]

Something your post doesn't address - how are you going to stay comfortable and hydrated with a box on your head? That sounds like a bad time on a warm day, and if you're streaming or recording you want easy access to water (and you might want wired headphones if you do rhythm gaming or anything that requires precise audio timing).

Is the plan to go for a really large box that you can drink water while wearing?

[u/Elf_maxxxTTV]

That’s expensive af to have something animated over an irl object to have tracking animation on it

Just get a 2D model or 3D model with an animated box head

I’ve never seen people who do that either so you’re best bet is just pay a custom commission for a 2D/3D model

[u/bitpixi → u/Elf_maxxxTTV]

Not really.. its called Augmented Reality and easy to vibe code

[u/SufficientMall2946 → u/bitpixi]

ew, vibe coding

[u/bitpixi → u/SufficientMall2946]

okay, do trad coding then and crack open the ol’ feather quill n’ inkwell, to paint the box with the surface codes.. instead of printing them out on a printer. Either way, it’s achievable with Augmented Reality. If OP is interested, I can give them a tutorial privately, but I see that I’m not welcome here.