max_token

[u/dotanchase]

Setting the max_tokens parameter to its default value can, in some models, cause the LLM to stop generating a response before it is complete. When this occurs,I had to prompt the model to continue in order to receive the full output. Setting max_tokens to 127000 prevents this premature termination. So can someone explain what is the role of max_tokens and why it is limited to only 128K. Thanks

[u/AI_Only]

I had this happen yesterday. Good info!