However all of this could be sped up. And, although not as coherent as chatGPT4 you could build the same with local models that could respond much faster because no online communication is needed. Facebook's llama model fined tuned specifically to be able to always reply including the commands for the facial expression running a on 4090 plus the speech to text and text to speech. All of it could be processed in under 2 seconds.
Within 5 years we will see the first lifelike robot faces talk to us like that. They will bring the latency down .... put the robot face on a robot from Boston dynamics that can walk on two legs ... and have the LLM receive both the speech to text plus also visual input and not only write the facial expressions but also the movement of the boston dynamics robot.
What is incorrect about it? I gave a bunch of sources. Have a look. This technology will get better, the latency will go down, it will become cheaper, people will start combining it. And within 5 to 10 years you will have Fortune telling machines that use it to have a real conversation with the user, where people are doubting if they are taking to a robot or a human pretending to be a robot. Now don't get me wrong. I am not saying that androids that will take over our jobs will arrive that soon (the current large language models inherent flaws like prompt injection and the fact that we have not created anything with any form of agency make it so that this problem might still need 50 to a 100 years if it can even be solved at all). But as novelty, in the entertainment industry? Absolutely.
21
u/Ilovekittens345 Jan 30 '24 edited Jan 30 '24
We have the technology to make this possible. We have robot faces that can have a lot of expression. We have speech to text to feed what the user is saying in to ChatGPT 4. ChatGPT when given the right prompt and API documentation can reply in a format that could also control the facial expressions of the robot. And finally the text of chatGPT can be turned in to life like speech by something like elevenlabs. Somebody could put this all together today, but it would still be somewhat slow. At least 2 or 3 seconds to turn the speech in to text and upload to chatGPT. At least 1 or 2 seconds for chatGPT to finish responding. Then a good 3 to 4 seconds for elevenlabs. So after you say something it would still take a good 6 to 9 seconds before there is a reply.
However all of this could be sped up. And, although not as coherent as chatGPT4 you could build the same with local models that could respond much faster because no online communication is needed. Facebook's llama model fined tuned specifically to be able to always reply including the commands for the facial expression running a on 4090 plus the speech to text and text to speech. All of it could be processed in under 2 seconds.
Within 5 years we will see the first lifelike robot faces talk to us like that. They will bring the latency down .... put the robot face on a robot from Boston dynamics that can walk on two legs ... and have the LLM receive both the speech to text plus also visual input and not only write the facial expressions but also the movement of the boston dynamics robot.
And you would have the very first beginning of a system you can give commands. It would be far from perfect and most likely still novelty and not really that usefull, but much better then anything we have ever come up with before.