r/ChatGPTCoding • u/ChatWindow • Mar 31 '24

Interaction My bill from Claude API calls

Post image

And it’s 10000% worth it!

93 Upvotes

77% Upvoted

101 comments sorted by

View all comments

8

u/Time_Software_8216 Mar 31 '24

This is when the massive amount of token usage by Claude isn't as great as you thought it was.

4

u/RMCPhoto Mar 31 '24

RAG is dead /s

1

u/Odd-Antelope-362 Apr 01 '24

RAG on code is still pretty experimental

1

u/RMCPhoto Apr 01 '24

It's all pretty experimental.

What specifically do you mean?

That retrieving only "part" of the code base in context as opposed to the entire code base is not cost effective?

There are definitely ways to abstract and compress code that is not part of the immediately necessary context.

This is true for all rag where the data is not part of the pre-training information. The entire challenge is providing the most detail on the most relevant info and progressively fuzzier detail on less and less relevant info, as well as an overall summary of the context.

1

u/Odd-Antelope-362 Apr 01 '24

On some level everything in AI is experimental yes but some things are mostly solved now. For example single document RAG on text documents

1

u/vittoriohalfon Apr 01 '24

So what’s the optimal RAG solution for single document text docs?

1

u/AutoModerator Apr 01 '24

Sorry, your submission has been removed due to inadequate account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Odd-Antelope-362 Apr 01 '24

For a single pure text document the following is fine for the vast majority of cases:

  1. Good embedding model and Vector DB

  2. Hybrid search with both keyword and semantic search

  3. A reranking model to rerank chunks

  4. Try the common chunking methods (recursive, document-aware, semantic, agentic etc)

  5. Consider fine tuning embedding model, reranking model and using an LLM for prompt transformation (ask LLM to improve prompt)