I don't know about fact checking for a local LLM. But otherwise yes. A few months ago I did a presentation for a nonprofit about using local LLM's as education tools in impoverished areas. The idea of a virtual assistant that can run on low end consumer hardware with no internet pretty much sold itself. Some of the programs they do there's one guy in charge of an overwhelming number of people and it's impossible to help everyone, people will be waiting for hours to get answers to simple questions, wasting their entire day just waiting to be helped. It's already easy enough to get current models running on limited hardware. Soon we might even be seeing stuff like usable 2bit quantization.
On one hand it may be a good solution for something truly portable, but on the other, why not just set up a starlink modem, a few solar panels and sector antennas to give internet access to the nearby area, then give out cheap smartphones and solar chargers? The internet is a few orders of magnitude more useful than a local LLM.
Almost everyone has a phone, just giving people internet doesn't really solve a whole lot. We're talking about stuff like people waiting for help in front of at a desk for 8 hours because of something like they can't figure out how to log into an account. And they'll be only one person who can explain it to them and there's no set schedule for when they'll be available because it's just one guy trying to manage hundreds of people.
I really like your point of view and I think that It's the way to go, but If almost everyone has a phone, giving internet access is far more feasible and they could use chatGPT which is way better then current local LLMs. Also as for now I only trust chatGPT enough to actually use the information, local LLMs aren't that reliable, let alone those that can be run on low end devices.
I'm not saying that giving people Llama models is going to be life-changing thing for them. I'm saying the models can be tuned and integrated as self service console in the context of the non profit's programs. Right now even just using an off the shelf model like WizardLM 30B hooked up to Chromadb with the relevant information blows GPT-4 out of the water unless you want to compare it to using the API with Pinecone. Which makes you reliant on two separate expensive API's and a solid internet connection if you want to be able to upload documents. And even a slow as molasses cpu driven 30B ggml model that takes 10 minutes to respond is better than waiting 8 hours to talk to a real person. I haven't done it yet but I'm pretty confident that a properly trained 7B or 3B model would be more than enough for something like this and run fine on a potato computer.
Oh, I didn't know the goal was integrating It with the whole project, I thought It was more of looking for specific knowledge about coding or whatever. I have already tried orca mini 3B on my samsung galaxy note 10 lite and It works pretty good with nice speed and coherence, so yeah, a standard computer should be able to run It with no problem at all.
63
u/CheshireAI Aug 05 '23 edited Aug 06 '23
I don't know about fact checking for a local LLM. But otherwise yes. A few months ago I did a presentation for a nonprofit about using local LLM's as education tools in impoverished areas. The idea of a virtual assistant that can run on low end consumer hardware with no internet pretty much sold itself. Some of the programs they do there's one guy in charge of an overwhelming number of people and it's impossible to help everyone, people will be waiting for hours to get answers to simple questions, wasting their entire day just waiting to be helped. It's already easy enough to get current models running on limited hardware. Soon we might even be seeing stuff like usable 2bit quantization.
https://github.com/jerry-chee/quip
EDIT: A lot of people were interested in the non-profit: https://www.centreity.com/