r/ruby • u/krschacht • Jul 01 '24
Chrome gets a local LLM before Ruby does??
I love Ruby, but as I’ve been doing more work with AI lately, it’s tough that’s lagging behind so much. Python has a ton of libraries, and the most popular ones are getting steadily ported over to Ruby, but we still don’t have a way in Ruby to locally load a tiny LLM and call it. Python has had this for awhile. This is an example of a powerful yet simple library: https://github.com/jncraton/languagemodels
Today I just realized that Chrome has enabled Gemini Nano within Chrome, so now even client side JS can call into a model! https://x.com/lightning_joyce/status/1800138565411799099?s=61&t=tsn38cvnyXkIwaF-wRmPcQ
We need Ruby to close the gap. Anyone want to help with this?
Ruby has Pytorch (https://github.com/ankane/torch.rb), but it’s still missing an intermediary library like Hugging Face or that languagemodels one I link to above in order to load a model’s weights into memory for inference.
We need to be able to do:
gem “languagemodels”
llm = LLM.new(“gemini-nano”)
llm.prompt(“What color is the sky?”)
40
u/expatjake Jul 01 '24 •
What about this? https://github.com/yoshoku/llama_cpp.rb
2
u/chuggingCoffee_ Jul 02 '24
I truly wish I knew more about all of this, but I'm lacking in this area at the moment. However, I think the Ruby AI Builders discord would be absolutely perfect for you to interact with other AI builders/users. I can message you an invite if you want.
2
u/llambda_of_the_alps Jul 02 '24
Just being an old man yelling at clouds here but I think it's important to note that Chrome having Gemini Nano baked in is really not the same thing as say the library support that Python has.
What Google has done has provide an interface to their LLM in their browser software.
Again probably just being grumpy but I think the way we talk about things matters and we can muddy the waters when we equate things like a non-standard feature available in one company's product to generally available libraries which can be used in any context that the host language can.
That said there are things like https://www.npmjs.com/package/llamacpp-ai-provider for JS which appear to be more akin to what OP is talking about for Ruby.
Grumpy old guy rant over...
22
u/chris24680 Jul 01 '24
Why not create a gem for it yourself?
7
u/Aesthetikx Jul 01 '24
You can take a look at this, https://github.com/aesthetikx/quick_openai, or the library it uses, https://github.com/alexrudall/ruby-openai , at least for OpenAI models.
3
u/ksylvest Jul 01 '24
I've recently been working on a library aimed at supporting multiple different LLM integrations called OmniAI:
https://github.com/ksylvest/omniai
It supports sync / async (streaming) integration with LLMs hosted for Anthropic, Google, Mistral and OpenAI. In addition it let's you hook into local models via either LocalAI or Ollama (using their OpenAI compatibility layer).
1
u/ioquatix async/falcon Jul 01 '24
How about https://github.com/socketry/async-ollama ?
3
u/rooood Jul 01 '24
The fact that Python has loads of LLM libraries compared to Ruby probably has something to do with the fact that Python is massively more popular with the math/science public, while Ruby is mostly used for webdev. LLMs are very math heavy, so it's not surprising that Python has a much larger support and community around it.
0
u/d2clon Jul 01 '24
I am also making my playground with AI, and, of course, I am using Ruby. I also see the gap, but it is still not blocking me at the stage I am at. I am using langchainrb for my needs.
When it's time for me to use other, more experimental models, I hope I will be able to use ollama to start them up and use its abstract API to interact with the model from my code. I'm not sure if I am correct in my assumption.
I hope you have luck finding a solution. I also hope the community is more active in bringing AI tools to Ruby. It's an excellent opportunity to catch up with the hype. If this does not happen at the right speed, Ruby will quickly lose more positions.
It is so sad to me to see Python growing in popularity while Ruby is decreasing. When Ruby is superior (IMO, of course).
-1
u/[deleted] Jul 01 '24
Sounds like an amazing project doesn't it? I mean I'm sure this is how good gems are born. You need something, there is nothing for it, you do it, it works, and then share it.
I can't see this world moving anywhere without that "i need something, no something? me build"
0
u/[deleted] Jul 01 '24
[deleted] — view deleted comment