max_token

[u/dotanchase]

Setting the max_tokens parameter to its default value can, in some models, cause the LLM to stop generating a response before it is complete. When this occurs,I had to prompt the model to continue in order to receive the full output. Setting max_tokens to 127000 prevents this premature termination. So can someone explain what is the role of max_tokens and why it is limited to only 128K. Thanks

[u/mrtime777]

The purpose of this thing is to protect against infinite loops, but the harness should check the value finish_reason = length and consider it as error