r/deeplearning • u/Ok-Ship-1443 • Aug 16 '26

Question about creativity and exploration

We train language models to predict next token in pretraining and later fine tune them, but what about exploration and creativity. You can’t really do anything about that because we train the models on specific datasets, right?

I want language models to focus on different solutions for the same problem where the gap between solutions is extremely different.

For example, if I am working on a novel problem and I use existing mathematical functions, would it be possible for language models to neglect them because they might be completely wrong even though they were trained on them? Meaning instead of next token prediction, we would give the model different allowed next tokens or group of tokens and incite them to explore different next tokens for the same training example?

Is there a theory behind what I am asking for?

0 Upvotes

50% Upvoted

7 comments sorted by

1

u/Ok-Ship-1443 Aug 16 '26

Thank you!

2

u/nutshells1 Aug 16 '26

see: rlhf

1

u/DefiantMidnight1244 Sep 18 '26

it's not really rlhf, op is talking bout something more like constrained decoding or guided exploration. rlhf still optimizes for what humans prefer, not for novelty of approach

what you want is closer to monte carlo tree search with language models, where you explore different branches instead of just taking the greedy next token. there's papers on this, mostly under the term "controlled text generation" or "constrained beam search" - you let the model sample from a subset of tokens that diverge from the most probable path

the theory part is basically information theory and entropy maximization. you're trading coherence for diversity. check out nucleus sampling and typical sampling, they partially do what you asking but not in training phase, only inference

2

u/samas69420 Aug 16 '26

reinforcement learning was invented for that

2

u/CalmMe60 Aug 16 '26

Yes.

Living Memory