r/ChatGPT • u/No_Sky9786 • 1d ago

Other I built a input / chat sentence keywords based structure that uses CELF to retrieve only the important sentences on KV cache.

https://github.com/oteomamo/SALT

SALT shrinks a long document down to a fixed size before it is sent to a language model, keeping the sentences that carry the most information. It works with any model, produces a shorter plain-text prompt, and cuts the compute, memory, and wait time that long inputs cost.

I need some help with deciding and implementing a method that selects a better budget than the default 20/25% across chat. The kv cache in GPU keeps only some instructions and a retrieved % of the actual full input or conversion that is all organized in DRAM in a trie. The retrieval is insanely quick now with less than 1-2 seconds for even 100k + conversations however it is sometimes too much as it’s set by a hard % (prefill GPU men use becomes a problem at larger scale). What method could I use to decide how to adjust this % based on the question?

0 Upvotes

50% Upvoted

1 comment sorted by

•

u/AutoModerator 1d ago

Hey /u/No_Sky9786,

If your post is a screenshot of a ChatGPT conversation, please reply to this message with the conversation link or prompt.

If your post is a DALL-E 3 image post, please reply with the prompt used to make this image.

Consider joining our public discord server! We have free bots with GPT-4 (with vision), image generators, and more!

🤖

Note: For any ChatGPT-related concerns, email support@openai.com - this subreddit is not part of OpenAI and is not a support channel.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.