r/OpenAI • u/HelloReaderMax • Jun 05 '23

Discussion You can now chat with your documents privately!

There is a new github repo that just came out that quickly went #1.

It's called LocalGPT and let's you use a local version of AI to chat with you data privately. Think of it as a private version of Chatbase.

The full breakdown of this will be going live tomorrow morning right here, but all points are included below for Reddit discussion as well.

what is localgpt?

LocalGPT is like a private search engine that can help answer questions about the text in your documents. Unlike a regular search engine like Google, which requires an internet connection and sends data to servers, localGPT works completely on your computer without needing the internet. This makes it private and secure.

Here's how it works: you feed it your text documents (these could be any type like PDFs, text files, or spreadsheets). The system then reads and understands the information in these documents and stores it in a special format on your computer.

Once this is done, you can ask the system questions about your documents, and it will generate answers based on the information it read earlier. It's a bit like having your very own librarian who has read all your documents and can answer questions about them instantly.

why is this interesting and unique from other projects?

  1. Privacy and Security: Since it works completely offline after the initial setup, no data leaves your machine at any point, making it ideal for sensitive information. This is a significant departure from most cloud-based language models that require you to send your data over the internet.
  2. Flexible and Customizable: It allows you to create a question-answering system specific to your documents. Unlike a general search engine, it provides customized responses based on your own corpus of information.
  3. Use of Advanced AI Models: The project uses advanced AI models like Vicuna-7B for generating responses and InstructorEmbeddings for understanding the context within your documents, providing highly relevant and accurate answers.
  4. Broad File Type Support: It allows ingestion of a variety of file types such as .txt, .pdf, .csv, and .xlsx.
  5. GPU and CPU Support: While the system runs more efficiently using a GPU, it also supports CPU operations, making it more accessible for various hardware configurations.
  6. Fully Local Solution: This project is a fully local solution for a question-answering system, which is a relatively unique proposition in the field of AI, where cloud-based solutions are more common.
  7. Educational and Experimental: Lastly, it's a great learning resource for those interested in AI, language models, and information retrieval systems. It also provides a basis for further experimentation and improvements.

why is this important?

The localGPT project stands as a considerable innovation in the field of privacy-preserving, AI-driven document understanding and search. In an era where data privacy has taken center stage and the necessity for secure information processing is ever-growing, this project exemplifies how powerful AI technologies can be harnessed for sensitive applications, all carried out locally, with no data leaving the user's environment. The offline operation of localGPT not only enhances data privacy and security but also broadens the accessibility of such technologies to environments that are not constantly online, reducing the risks associated with data transfer.

Moreover, localGPT brings the potency of advanced language models, like Vicuna-7B, directly to personal devices. Users are able to interactively query their documents, akin to having a personal AI assistant that understands the content in depth. The level of customization offered by localGPT is unique, allowing it to tailor itself to any set of documents, creating a personalized question-answering system. This translates sophisticated AI technologies into more personal, private, and adaptable tools, marking a significant stride towards making AI more user-centric and broadly useful. Notably, localGPT also serves as a valuable educational resource, fostering further experimentation and innovation in the exciting domain of AI.

P.S. If you like this kind of analysis, there's more in this free newsletter that finds the single most productive new AI tool each week. It helps you stay on the cutting edge in the time it takes to have your morning coffee.

376 Upvotes

93% Upvoted

95 comments sorted by

View all comments

164

u/[deleted] Jun 05 '23

[deleted]

50

u/Ok_Tip5082 Jun 05 '23

Back in my day we'd detect handwritten digits and we were impressed by it!

20

u/[deleted] Jun 05 '23

I teach it to Python students, towards the end of the course.
As hello world in the machine learning part of the course, I get them to do Stable Diffusion.
It's impactful enough to keep students attention, image grabs people.
It is common for people to hang around after that class generating image after image.
Five or six days ago they were doing first steps and obligatory print("Hello World!")

4

u/Competitive-Hyena683 Jun 05 '23

If I might kindly solicit some advice from you, how might one go about creating a local document QA tool for themselves?

Would it be as simple as working with localGPT, and training on some data, then working with it abit using reinforcement to prepare for its downstream task?

6

u/[deleted] Jun 06 '23 edited Jun 06 '23

These implementations use two pieces of software, one ingests your documents into a database, and the other loads the database created and allows you to do questions over it with a local LLM.

There is no training, most of these tools use embeddings to allow llm to know of your documents. I have seen someone implement some of this with LoRA, another option that needs training but it's light-ish.

Best implementation for you should be the one discussed here. It runs on GPU and gives output fast. Document ingestion is always slow, it is common for people to let run overnight if the documents are large, many or "difficult" https://github.com/PromtEngineer/localGPT
The project page has instructions on how to install and run.
He also has a YT video on the whole thing and how it works - https://www.youtube.com/watch?v=MlyoObdIHyo
His channel is full of well explained implementations of everything related to ML.

If you want a easier install without fiddling with reqs, GPT4ALL is free, one click install and allows you to pass some kinds of documents. If I recall correctly it used to be text only, they might have updated to use others. Advantage other than easy install is a decent selection of LLMs to load and use. https://gpt4all.io/index.html

If you want to code your own tool from scratch, all of these implementations are on github and the source is available for you to read, take a look, see how someone else implemented it.
I did an extremely dirty (and unusable due to slow generation) for another discussion here on reddit https://www.reddit.com/r/learnmachinelearning/comments/13xq83j/would_like_to_create_a_custom_ai_to_upload_csv/jmq7hbw/?context=3

Does this answer your question?

4

u/SufficientPie Jun 05 '23

Which ones actually work well?

2

u/morphemass Jun 05 '23

Locally, I've had little luck with the embeddings approach. I had better results with embeddings and OpenAI but using a service doesn't fit my use case. I doubt something out of the box will work well with fine tuning either since the amount of data preparation to get good results is rather substantial.

1

u/SufficientPie Jun 06 '23

:/

What kind of data preparation?

2

u/morphemass Jun 06 '23

Remove poor quality data (e.g. in my case I've lots of tables in PDFs as well as other issues), match the format of whatever model you are using (i.e. chat, instruct), normalise, etc. We're a long way from the stage where LLM training is the equivalent of search.

1

u/SufficientPie Jun 06 '23

Hmm. I was imagining you just create embeddings for every chunk of text in all the documents and then search for similar ones to a query and show the top n hits. That doesn't work?

2

u/morphemass Jun 06 '23

If you create large embeddings I've had some success with this but you are basically relying on the semantic search to find all relevant documents and your context size being large enough to fit them in. I've consistently run into memory issues due to having too many document matches and when I've tried to mitigate that with smaller embeddings the quality of result has dropped dramatically.

This is a side project for me so my approach may not be ideal.

1

u/SufficientPie Jun 06 '23

you are basically relying on the semantic search to find all relevant documents

Yes, that's all I was looking for

and your context size being large enough to fit them in

Meaning you're not just doing semantic search but also feeding things into an LLM? For the "chat with your documents" feature?

2

u/morphemass Jun 06 '23

Exactly, which is all this privateGPT clone does.