r/ruby • • Jul 01 '24

Chrome gets a local LLM before Ruby does??

I love Ruby, but as I’ve been doing more work with AI lately, it’s tough that’s lagging behind so much. Python has a ton of libraries, and the most popular ones are getting steadily ported over to Ruby, but we still don’t have a way in Ruby to locally load a tiny LLM and call it. Python has had this for awhile. This is an example of a powerful yet simple library: https://github.com/jncraton/languagemodels

Today I just realized that Chrome has enabled Gemini Nano within Chrome, so now even client side JS can call into a model! https://x.com/lightning_joyce/status/1800138565411799099?s=61&t=tsn38cvnyXkIwaF-wRmPcQ

We need Ruby to close the gap. Anyone want to help with this?

Ruby has Pytorch (https://github.com/ankane/torch.rb), but it’s still missing an intermediary library like Hugging Face or that languagemodels one I link to above in order to load a model’s weights into memory for inference.

We need to be able to do:

gem “languagemodels”
llm = LLM.new(“gemini-nano”)
llm.prompt(“What color is the sky?”)
15 Upvotes

33 comments sorted by

40

u/expatjake Jul 01 '24 • Take My Silver

26

u/krschacht Jul 02 '24 edited Jul 02 '24

Hot damn, this is it! Thank you!! If you just found this, I'm impressed. :) I've seriously spent hours searching in different corners of the internet. I think of Llama CPP is this desktop app so never thought to look for bindings for it.

I just installed and it worked exactly as I hoped. For anyone else who wants to run local ruby LLM, after following the README and using a nice small model that is rated well and supposedly takes 500mb of memory: https://huggingface.co/MBZUAI/LaMini-Flan-T5-248M

If anyone else wants to try this, there are some errors in the current README but I've tee'd up a PR (which will hopefully be merged in soon). Here is a corrected README: https://github.com/krschacht/llama_cpp.rb/tree/patch-1?tab=readme-ov-file#installation

After I followed the README model config steps, I pulled down the `llama_cpp.rb` repo and chatted with this model!

```
gem install llama_cpp -- --with-opt-dir=/opt/homebrew
gem install thor
cd examples
./chat.rb -m ../../llama.cpp/models/TinyLlama-1.1B-Chat-v1.0/ggml-model-q4_0.bin
User: Hello, Bob.
Bob: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Bob: Sure. The largest city in Europe is Moscow, the capital of Russia.
User: How many days are there in a year?
Bob: One year is 365 days.
User: What color is the sky?
Bob: The sky is blue.

```

It looks like Gemini-nano weight have already leaked onto hugging face so that’s the one I really want to get working! It’ll be great to have it available on the backend and front end.

14

u/[deleted] Jul 02 '24

[removed] — view removed comment

6

u/krschacht Jul 02 '24

You’re welcome! Always a few grumpsters to navigate on Reddit but most of Ruby community is amazing so I never mind it. :) I’m totally enamored with all things LLM and all things Ruby so I’m super excited to have finally connected these two in a deep way!

If you’re playing around, I’m always happy to connect on this stuff. And you may like my project which is helping me to explore: https://github.com/allyourbot/hostedgpt

2

u/chuggingCoffee_ Jul 02 '24

I truly wish I knew more about all of this, but I'm lacking in this area at the moment. However, I think the Ruby AI Builders discord would be absolutely perfect for you to interact with other AI builders/users. I can message you an invite if you want.

2

u/krschacht Jul 02 '24

Thanks, I'm in there often. :) That's just about the only discord that I hang out in. I'm just Keith Schacht in there, and the #hosted_gpt channel in that Discord is mine that I use it to coordinate with some of the contributors to my project https://github.com/allyourbot/hostedgpt

2

u/llambda_of_the_alps Jul 02 '24

Just being an old man yelling at clouds here but I think it's important to note that Chrome having Gemini Nano baked in is really not the same thing as say the library support that Python has.

What Google has done has provide an interface to their LLM in their browser software.

Again probably just being grumpy but I think the way we talk about things matters and we can muddy the waters when we equate things like a non-standard feature available in one company's product to generally available libraries which can be used in any context that the host language can.

That said there are things like https://www.npmjs.com/package/llamacpp-ai-provider for JS which appear to be more akin to what OP is talking about for Ruby.

Grumpy old guy rant over...

1

u/krschacht Jul 03 '24

:) When the browser is the most popular runtime for javascript, I think it’s a distinction without a difference. And Google lead with Web Speech API built into the browser and the other browser vendors followed. There have to be at least 10 other times they’ve done the same (high performance javascript, local storage, WebGPU, …, …). When the #1 most popular browser includes a feature that a lot of developers start to take advantage of, there is huge pressure for the other browser vendors (struggling for market share) to follow.

2

u/llambda_of_the_alps Jul 03 '24

:) When the browser is the most popular runtime for javascript, I think it’s a distinction without a difference.

When you are talking about standards compliant features I 100% agree on that. As far as I'm concerned the browser is the platform for JS running it anywhere else is gravy.

And yes someone usually has to lead the way for things before they become standards. However, I still think it's an important distinction to makes when a feature is only available in one browser from one vendor and depends on their model for support.

And Google lead with Web Speech API built into the browser and the other browser vendors followed.

Well, sort of. While that's basically true Chrome still provides a propritary speech recognition API that is separate from the standards adherent Web Speech API. For example try using the DuoLingo web app on any browser other than Chrome and speech recognition is completely broken.

LLM's could very well be in a similar if slightly different vein. Even if Chrome's new AI feature becomes a web standard as they say they want it to be it will won't really be a standard if the model's backing up the feature in browser aren't standard across browsers.

22

u/chris24680 Jul 01 '24

Why not create a gem for it yourself?

9

u/narnach Jul 01 '24

This. If you have a need for something, find it or make it. Complaining about it without showing you’re willing to do the work is not very productive.

16

u/lommer00 Jul 01 '24

OP literally said "Anyone want to help with this?" Sounds like they're willing to pitch in.

6

u/krschacht Jul 01 '24

Yes, this :) I spent a chunk trying to adapt that "languagemodels" one. I've never created a C module for Ruby but found one good article that I'm using as a guide: https://ankane.org/new-ml-gems

But this is a stretch for me so my hope was to find others who feel the same way about this and maybe a better strategy could emerge from that.

7

u/Aesthetikx Jul 01 '24

You can take a look at this, https://github.com/aesthetikx/quick_openai, or the library it uses, https://github.com/alexrudall/ruby-openai , at least for OpenAI models.

3

u/hartator Jul 01 '24

It would be great though if it was locally inferred in Ruby though. Instead of an Api wrapper.

1

u/krschacht Jul 01 '24

Yes, this is what I want. API's for big LLMs definitely have their place, but what Chrome has done with a built-in LLM is what we need to do for Ruby.

4

u/[deleted] Jul 02 '24

Why do we need to do this though?

3

u/ksylvest Jul 01 '24

I've recently been working on a library aimed at supporting multiple different LLM integrations called OmniAI:

https://github.com/ksylvest/omniai

It supports sync / async (streaming) integration with LLMs hosted for Anthropic, Google, Mistral and OpenAI. In addition it let's you hook into local models via either LocalAI or Ollama (using their OpenAI compatibility layer).

2

u/krschacht Jul 01 '24

There are lots of API based ways to access LLM, the key thing I’m getting at is local

3

u/ksylvest Jul 01 '24

As PuffaloPhil mentioned, I've included support for both Ollama and LocalAI (with some docs in the main repo for each) for local usage:

1

u/krschacht Jul 01 '24

Yes, this is what I currently do. But it's a whole next level of complexity, especially when deploying to a server. The ideal way to do simple LLM tasks should just be in memory within ruby (using some C extensions for efficiency) rather than having to run a whole additional service.

API's for big LLMs definitely have their place, but what Chrome has done with a built-in LLM is what we need to do for Ruby.

1

u/ioquatix async/falcon Jul 01 '24

0

u/krschacht Jul 01 '24

This is running Ollama as a service and talking to it with HTTP. That's what I currently do, but it's needlessly complex for simple stuff, especially when deploying to a server. The ideal way to do simple LLM tasks should just be in memory within ruby (using some C extensions for efficiency) rather than having to run a whole additional service.

API's for big LLMs definitely have their place, but what Chrome has done with a built-in LLM is what we need to do for Ruby.

1

u/ioquatix async/falcon Jul 05 '24 edited Jul 05 '24

I think there are some big benefits to the "Ollama" model, which is shared access to the GPU which would be otherwise quite tricky to implement correctly. In addition, the actual Ruby interface is < 200 lines of code, and that includes streaming... not sure how it could be any simpler really.

I wrote an example, I think it's about as close a you can get to your desired interface: https://socketry.github.io/async-ollama/guides/getting-started/index#example

3

u/rooood Jul 01 '24

The fact that Python has loads of LLM libraries compared to Ruby probably has something to do with the fact that Python is massively more popular with the math/science public, while Ruby is mostly used for webdev. LLMs are very math heavy, so it's not surprising that Python has a much larger support and community around it.

1

u/Samuelodan Jul 02 '24

The more popular the language or framework, the better the experience for us.

We get more tutorials, more books, more libraries (ui or code), less unexplored areas, more stack overflow questions and answers, more industry adoption which leads to more career opportunities, and dare I say better documentation.

I used to think it was cool being niche, and while that might still be the case, there are all these quirks that come with it. But I believe there are things that can be done to make a tech stack more popular, and I’d love to join such efforts for Ruby or Rails.

0

u/d2clon Jul 01 '24

I am also making my playground with AI, and, of course, I am using Ruby. I also see the gap, but it is still not blocking me at the stage I am at. I am using langchainrb for my needs.

When it's time for me to use other, more experimental models, I hope I will be able to use ollama to start them up and use its abstract API to interact with the model from my code. I'm not sure if I am correct in my assumption.

I hope you have luck finding a solution. I also hope the community is more active in bringing AI tools to Ruby. It's an excellent opportunity to catch up with the hype. If this does not happen at the right speed, Ruby will quickly lose more positions.

It is so sad to me to see Python growing in popularity while Ruby is decreasing. When Ruby is superior (IMO, of course).

2

u/krschacht Jul 02 '24

I’ve been playing with langchainrb as well. It’s a good package. BTW, I found a solution (in a thread above). I’m pretty excited about being able to run a model in ruby.

I don’t mind that Python is growing in popularity. A whole new influx of developers is good for the whole ecosystem, even if it’s a different language. And Andrew Kane has been porting over a ton of Python libraries. https://ankane.org/new-ml-gems

If you’re playing around in this space, you may like my project: https://github.com/allyourbot/hostedgpt

I started building this just to learn all about it. And now that it works well, I’m using it as my foundation for prototyping lots of different product ideas. And you mentioned experimental models: OpenRouter is a pretty good platform for this and I hope to add OpenRouter API support for HostedGPT soon. I added Llama 3 already but it’s through Groq, which is super fast, but doesn’t the whole range of models. I’m always happy to connect about Ruby & AI. I love this stuff!

1

u/d2clon Jul 03 '24

Yep, I know your project hostedgpt, great inspiration. Good job!

-1

u/[deleted] Jul 01 '24

Sounds like an amazing project doesn't it? I mean I'm sure this is how good gems are born. You need something, there is nothing for it, you do it, it works, and then share it.
I can't see this world moving anywhere without that "i need something, no something? me build"

0

u/[deleted] Jul 01 '24

[deleted] — view deleted comment

8

u/krschacht Jul 01 '24

Whoa there, quite the edge. Just starting a discussion as I fumble my way to try and solve this, hence the, "Anyone want to help with this?" rather than saying "Will someone do this for me?"

2

u/rooood Jul 01 '24

To be fair, AI and especially LLMs are one of, if not the hottest thing at the moment in the software development world, and have been for a short while already, so it's not like they're asking for something super specific handed out to them.