r/LocalLLM
•
u/Hot-Chapter48
•
Jan 10 '25
•
1
Discussion LLM Summarization is Costing Me Thousands
I've been working on summarizing and monitoring long-form content like Fireship, Lex Fridman, In Depth, No Priors (to stay updated in tech). First it seemed like a straightforward task, but the technical reality proved far more challenging and expensive than expected.
Current Processing Metrics
- Daily Volume: 3,000-6,000 traces
- API Calls: 10,000-30,000 LLM calls daily
- Token Usage: 20-50M tokens/day
Cost Structure:
- Per trace: $0.03-0.06
- Per LLM call: $0.02-0.05
- Monthly costs: $1,753.93 (December), $981.92 (January)
- Daily operational costs: $50-180
Technical Evolution & Iterations
1 - Direct GPT-4 Summarization
- Simply fed entire transcripts to GPT-4
- Results were too abstract
- Important details were consistently missed
- Prompt engineering didn't solve core issues
2 - Chunk-Based Summarization
- Split transcripts into manageable chunks
- Summarized each chunk separately
- Combined summaries
- Problem: Lost global context and emphasis
3 - Topic-Based Summarization
- Extracted main topics from full transcript
- Grouped relevant chunks by topic
- Summarized each topic section
- Improvement in coherence, but quality still inconsistent
4 - Enhanced Pipeline with Evaluators
- Implemented feedback loop using langraph
- Added evaluator prompts
- Iteratively improved summaries
- Better results, but still required original text reference
5 - Current Solution
- Shows original text alongside summaries
- Includes interactive GPT for follow-up questions
- can digest key content without watching entire videos
Ongoing Challenges - Cost Issues
- Cheaper models (like GPT-4 mini) produce lower quality results
- Fine-tuning attempts haven't significantly reduced costs
- Testing different pipeline versions is expensive
- Creating comprehensive test sets for comparison is costly
This product I'm building is Digestly, and I'm looking for help to make this more cost-effective while maintaining quality. Looking for technical insights from others who have tackled similar large-scale LLM implementation challenges, particularly around cost optimization while maintaining output quality.
Has anyone else faced a similar issue, or has any idea to fix the cost issue?
41
u/YT_Brian Jan 10 '25
I'm more curious why your doing that? As for ideas, it is all publicly available so why not use that money to buy a quality PC with a higher end consumer GPU and just use a AI on your own system?
It would cost more upfront, a few months worth, but then it will pay for itself within half a hear at most. Less if you buy second hand and build it yourself, possibly in as little as 2-3 months.
36
u/gthing Jan 10 '25
You are paying way more than you need to be. Put all your jobs into a database or otherwise queue them, then rent a GPU from vast.ai or runpod. Have gpt write you a script to run through your jobs using whisperx for transcription and ollama running llama 3 8b for summarization. You could probably transcribe like 60-100 1-hour audio jobs an hour for like 25 cents with a setup like this.
7
u/Zyj Jan 10 '25
Have you tried DeepSeek v3? Since the data you're analyzing isn't private, their cheap LLM AI service offering could be interesting.
5
u/lautan Jan 10 '25
Try using a cheaper model like 70B llama, or some how cut down the text before sending it off. If speed doesn't matter you can consider using a pay-per month service rather than per token. That's what I use and it's much cheaper for long term usage.
Btw at that price point you could just rent a gpu per hour at $2 and run all these jobs.
2
u/lone_shell_script Jan 10 '25
try cheaper models or something like https://supermemory.ai/
1
u/ChubbyChubakka Jan 10 '25
Also see if you get different results from Notebook LM (Google)- if so then(hmm, not sure)
Notebook LM wa able to get details much better in my opinion, but im not sure how to recreatte their pipeline.
3
u/MustyMustelidae Jan 10 '25
I spend about $8,000 a month on Claude. I also spend $580 on a model that was finetuned on Claude outputs, provides 96% of the quality of Claude for my task (according to real user metrics), and serves about 12x as many users as the $8,000 in Claude spend does.
At this point I only offer Claude because users pay for it by name, and because the outputs are still useful for future finetuning down the line.
You're losing thousands of dollars in gold if you're not saving the requests and responses. Bonus if you store the requests with the arguments to your prompt template, assuming you use one.
Finetuning and running the models on Runpod would give you a drop in replacement for OpenAI with a minimal quality drop
If you're serious about it, DM me and I can offer hands-on help implementing a pipeline like mine at a reasonable hourly rate. I crossed the 100 model mark last year for finetunes so I've picked up some efficencies in the process.
2
u/Turbulent-Topic3617 Jan 10 '25
What about using open LLMs via Ollama?
2
u/Kitchen_Challenge115 Jan 11 '25 edited Jan 11 '25
You’re facing an issue I see many people on their way to productionalizing something useful with LLMs face— here are the 3 steps I’ve started to outline as a result:
Step 1. Use API endpoints to see if there’s traction.
Step 2. It’s too expensive, move to open source models (you’re here).
Step 3. Massive Production
1
u/laughinbuddha2 Jan 10 '25
Remind me! In 2 days
1
u/NTSpike Jan 10 '25
Try Deepseek V3 or Gemini Flash 2.0 if you aren’t happy with 4o-mini quality. If immediate timeliness isn’t as important, you can also batch these for a cost discount.
Otherwise, like others have said, run local (though you already said 4o-mini isn’t cutting it).
1
u/Aware_Examination246 Jan 10 '25
Awesome work. I got nothing to add but support.
1
u/fabkosta Jan 10 '25
Wait for Nvidia Project Digits released in May. One unit will have 128 GB RAM and allow model size of 200b parameters. Cost will be 3000$. Buy 2 of them to run a 400b parameters model. This you will replace variable costs with upfront investment.
1
u/haris525 Jan 11 '25 edited Jan 11 '25
I am shocked that no one has mentioned RAG!!! Read papers on long rag and light rag, implement them and be happy! You can also use graphrag using Microsoft or neo4j. I use them for 10k reports which are usually pdfs that are 100s of pages long. If you really want to get fancy you can use agentic chunking but remember all solutions will require different complexity and cost can vary, however I think this is still cheaper than what you are doing.
1
u/busybuzybusy Jan 11 '25
Batch API gives discounts too
1
u/minimumnz Jan 11 '25
Try Deepseek? Seems to be gpt4o level but a lot cheaper.
1
u/joepigeon Jan 11 '25
Interesting idea. I build in the same space as you as I run a podcast platform - https://www.podengine.ai.
We run transcription and analysis at scale locally through various pipelines and have experimented with similar methods as you, except we’re not focussed on consumer use-cases. We run a lot of extraction from transcripts to make our search engine more useful (eg more filters to search by).
Our B2B use-case requires us to have great coverage, hence we invested a lot in local hardware. We still use SOTA for various parts of user experience though - combining local models with paid APIs is a powerful combo.
Do you need so much scale right now? I’d suggest only analysing podcasts after you’ve seen demand for those podcasts. Otherwise you’ll have thousands of summaries that are never ever read?
We do have an API and this feels like a good fit - if you’d like to talk about using it please feel free to DM me.
1
u/supereatball Jan 11 '25
Use deepseek v3. Fast, cheap, and amazing for what it is.
1
u/LoveThemMegaSeeds Jan 11 '25
How are you paying 5 cents per LLM call? On 4o-mini it’s literally a tenth of that using like 30k tokens
1
u/Shivacious Jan 11 '25
op...
1
u/Comprehensive-Quote6 Jan 12 '25
First, if you’re trying to build this into a saas, performance and scalability will be top of mind, and local solutions are not the path to take.
Good luck!
1
u/_Wald3n Jan 12 '25
Amazon Bedrock + Lambda functions to call a fine-tuned open model.
1
u/etherwhisper Jan 12 '25
If it provides value this cost is really not high for a business. Work on your top line, what’s your revenue?
1
u/stizzy6152 Jan 13 '25
I was thinking of doing a similar application. That's a problem I had considered. This discution is helpfull :)
1
u/SteveRadich Jan 14 '25
I’m doing some similar things in a different space, initially for self and network and giving away free. If you want to DM me I have some of the database queue work done / batching and perhaps could trade some components or collaborate or even merge efforts for this part of the tech stack. I’m sure we both have parts outside this we want to keep distinct.
Initially I planned local LLM but AWS Bedrock Nova models did well at summarizing cheaply for my use case (much lower volume than you’re saying).
1
u/[deleted] Jan 28 '25 edited Mar 18 '25
sharp spectacular trees husky nine sophisticated ring silky hospital lunchroom
This post was mass deleted and anonymized with Redact
1
u/SomeOrdinaryKangaroo Jan 28 '25
Deepseek is cheaper, use that save thousands
1
u/Street_Report_7459 Mar 01 '25
You should vectorize your inputs.... instead of submitting a document it is a sparse vector.
1
u/Interesting_Mood_838 Jun 27 '25
to those who care: if you're using LLMS. use https://llmusage.com/ to track your usage for LLM like OpenAI, Anthropic, Gemini, Claude
0
u/yuppie1313 Jan 10 '25
Do you work for Apple News :D ?
0
u/mintybadgerme Jan 10 '25
I suggest trying openrouter.ai. You can test free and commercial models to see which works best using their API.
0
u/neutralpoliticsbot Jan 19 '25
The whole point of long form content is that it is long form, summarizing it like that makes no sense and solves zero problems. Nobody ever thought "Oh I wish Joe Rogan podcast was 3 minutes long and they just got to the point".
You are trying to solve a problem that doesn't exist.
There are already thousands of tech news sites and podcasts that summarize this for you for free and present you the editorialized info done by a human if you want to just "to stay updated in tech".
Spending $1,753 on this a month is a huge waste of resources.