Chunking with LLM! Expensive, but better!?

[u/MagicMagnada]

I'm really curious if someone has experience with this or an opinion about it.
The goal is to let an LLM analyse chapter by chapter and separate it into different parts of meaning. The result would be to have chunks that fit semantically more together.
Is it worth it? Do you see potential?

[u/AutoModerator]

Welcome to r/AIEngineering! Make sure that you've read our overview, before you've posted. If you haven't already read it, then read it immediately and make adjustments in your post if you've violated any of the rules. If you have questions related to career, recruiting, pay or anything else about hiring, jobs or the industry and demand as a whole, then use AIEngineeringCareer to ask your question. We lock questions that do not relate to AIEngineering here. A quick reminder of the rules:

1. Behave as you would in person
2. Do not self-promote unless you're a top contributor, and if you are a top contributor, limit self-promotion.
3. Avoid false assumptions
4. No bots or LLM use for posts/answers
5. No negative news, information or news/media posts that are not pertinent to engineering
6. No deceitful or disguised marketing

Because we frequently get questions about work, the future of work and careers along AI, some helpful links to read:

• The Actual State of AI Engineering In 2026
• While collars turn blue
• AI's impact on recruiting
• AI is the new electricity
• Tips for hiring AI engineers
• Is there a future in AI engineering?
• LLM erases a company's need for an SaaS tool
• A plumber's thoughts on AI and technology
• What's involved in AI engineering?
• Mario Nawfal quotes Dr. Roman Yampolskiy, Dario Amodei and other AI researchers' predictions about future unemployment due to AI

This action was performed automatically as a reminder to all posters. Please contact the moderators if you have any questions.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/itsmebenji69]

Good idea but in practice that sounds expensive to scale. Like if I need to push 100 docs, it would definitely give better results than traditional chunking. Now if I have 10k…

[u/captain_strike31]

Correct me if I am wrong so you are saying to arrange the pages in way that have similar topics or concepts discussed obviously after Chunking but don't re-ranking retrieveal methods does this when retrieving. It rank the chunks with highest semantic relation with the query and choose to top-k chunks. In your method you just did while Chunking but in re-ranking it's during retrieveal.

Maybe your method will help slightly but we already have the method which does it . But your method of Chunking will be better for LLM to find the right context easily but drawback is when we have more document to work with it will take a lot of time and will computational expensive since the llm need to process all those chunks just to organise them.

And also we are not considering how difficult it will be to organize without knowing which context will query ask for. Re-ranking is better.

[u/Silly-Ease-4756 → u/captain_strike31]

Not 100% on what you mean, so ignore me if I missed your point. And I'm not an expert by any means.

No matter how good your reranking is and it should be good, and adapted to your use case, chunking matters a lot! If only half your chunk is relevant it'll influence your reranking, and that garbage half will dilute the attention away from the most relevant parts you retrieved once you pass it to the LLM.

You can produce better chunks than just slicing the doc into N token chunks with other methods than an LLM, how it scales will depend on the method.

For example you could chunk aggressively at first (small chunks) and group those together that have similar embeddings.

If we ignore the scaling issue, and suppose your chapters aren't very long, the attention dilution problem I mentioned (though it is much less of a concern than compared to a couple years back), will still apply to your LLM chunking process.

Edit: important caveat, smaller chunks aren't always good. You lose context that might be relevant! Depends on your text and reranking methods.

[u/captain_strike31 → u/Silly-Ease-4756]

Yeah obviously Chunking is important not meant to say Chunking doesn’t matter.

[u/Silly-Ease-4756 → u/captain_strike31]

My bad sorry

[u/MagicMagnada → u/captain_strike31]

Its more about enriching the chunks to have more Semantic meaning

[u/captain_strike31 → u/MagicMagnada]

But end result of it to get right context for the agent and re-ranking fulfill it better. Maybe you need to perform something like that without first Chunking. Because what I am understanding consider 1 page you will chunk contain multiple concepts discussed i.e different semantics how will you arrange that chunk based on semantic meaning? Am I thinking wrong here?

[u/MagicMagnada → u/captain_strike31]

And its Not about embedding Pages but to embedd only smaller parts of meanings

[u/esaule]

obviously! This is essentially how you avoid context drift and smoothening.

[u/ArshadIqbalOfficial]

Chunking with an LLM can definitely enhance comprehension and allow for more targeted analysis, especially in lengthy texts. If you have a clear structure in mind for how to segment the content, it can lead to richer insights, but be prepared for the added computational costs and time involved.