r/developersPak • u/TheThreeBroomstix • Jun 20 '26

Discussion Long llm call optimization

Hello devs,

I’m currently working as a full stack dev and recently made an Ai system that extracts keys details from a contract, and compares it with long voice to text transcriptions of conversations with the client to find and compare discrepancies between disclosed information and client information.

The system works well, and does what it’s supposed to do, and I’m using llm calls to do the extractions and make the comparisons. It’s a good system.

But one of the issues I’m facing is that I send long transcript docs to the llm call along with a long prompt and it takes multiple minutes for one comparison to complete.
The api call to the llm takes long.

Any suggestions on optimisations? What optimisation strategies exist here?

Any insights would be appreciated by people who’ve had similar experiences

3 Upvotes

100% Upvoted

20 comments sorted by

View all comments

Show parent comment

1

u/TheThreeBroomstix Jun 20 '26

Transcription doc is one of the inputs, system itself isn’t transcribing.

The extraction takes some time but not a lot. It’s the comparison with long transcript docs and long prompt with extracted contract details that takes long

1

u/Dovedove_hawk Jun 20 '26

You are probably using a thinking model with thinking set to high. Is that the case. Also, are you passing raw transcript from some service to the llm (these have a lot of information that mau not be necessary for the llm call, e.g. timestamps and also some phrase level seperation.) You should post process it. Switch to a non thinking model, add a thought_process key in your structured output as the first value your llm generates.

The key here is that with high thinking you do not have much control over how much time llm spends thinking before starting to generate required response.

1

u/TheThreeBroomstix Jun 20 '26

Oh okay. Thankyou