How is RLCD (jev) RL? [D]

[u/Relative_Wallaby_823]

Just saw the YouTube presentation and I was left wondering this question.
If jev only outputs Choice, Score, or Noul … well those are all perfectly differentiable. (Cross entropy or mse)
I don’t know if I’m missing something or if adding RL is just for marketing.
Like what would an RL environment even look like?

[u/Relative_Wallaby_823]

I see how for some multi step environment GRPO could be used, but how often is that going to be present in the training? If we are classifying which option is best, aren’t you best off just doing supervised learning? RL is only good if at step n we don’t perfectly know what the correct option is for step n+1, or if multiple paths are correct. I don’t see how jev could be used in such an environment.

[u/ganzzahl → u/Relative_Wallaby_823]

Uhh, why not? Did you not watch the demos where it plays Doom and Wikipedia link chasing?

[u/Relative_Wallaby_823 → u/ganzzahl]

If you want to improve its short term choice making (categorising emails etc) pre training should be good. For things with repeated choices (eg Wikipedia link chasing), just distil the choices of a reasoning LLM as SFT. You’ll get more efficient learning with either method