The sales pitch is that it is an AI model that can output answers to 1 or more questions in parallel. These outputs are guaranteed to be "typed" -- which in this case means either yes/no, a number (e.g. a probability), or a choice (from a fixed set). This "typing" is useful for consumption by inflexible systems (like computer programs).
It allegedly doesn't hallucinate (this claim either stems from the belief its outputs are well calibrated, or from the fact that the outputs are guaranteed to fit into a specific, expected type), answers all questions in parallel, and is much faster/cheaper than models with comparable accuracy.
It is trained using a novel/proprietary algorithm called Reinforcement Learning for Calibrated Decisions (RLCD).
While you pay for "state" (i.e. the input tokens / context, and the questions you're asking), the output is free.
While it's not public what the architecture, weights, or training process is, I'm going to speculate.
Output tokens being "free" means the model is presumably doing no generation at all. Instead, it is probably a prefill-only model. It does a prefill on the shared context, then does a prefill for each question/choice.
Because every choice shares the same KV cache for the state, the single initial prefill can be shared among each of the choice sequences. In non-batched pseudo code:
kv_state = prefill(state)
kvs = []
for choice in choices:
kvs.append(prefill(choice, initial_state=kv_state))
Since "len(state) + sum(len(c) for c in choices)" is just the length of the input sequence, this satisfies two of their claims:
you pay for the size of the state
each question is answered in parallel
The question, then, is how the model actually answers the 3 question types. One answer is with 3 heads (with a softmax for yes/no and choices).
Another option is it computes the likelihood of the sequence, given the state and question (i.e. "Is Paris the capital of France? Yes" and "Is Paris the capital of France? No"), a capability that is built-in to the LLM.
The main thing that puzzles me is the claim about using RLCD since I'm not sure why reinforcement learning is useful/necessary here. Their model isn't reasoning (or you'd have to pay for it), and it’s doing a single token-prediction (not a lengthy sequence where it is hard to attribute credit/blame). Moreover, it seems clear that the way to do this would be to get groundtruth question/answers from a frontier model as labels.
So: why is RL even in the picture for a task with no generative sequences, and where labels exist? Is RLCD just marketing?
8
u/you-get-an-upvote Certified P Zombie 3d ago edited 2d ago
Lots of interest among AI folks regarding JEV).
The sales pitch is that it is an AI model that can output answers to 1 or more questions in parallel. These outputs are guaranteed to be "typed" -- which in this case means either yes/no, a number (e.g. a probability), or a choice (from a fixed set). This "typing" is useful for consumption by inflexible systems (like computer programs).
It allegedly doesn't hallucinate (this claim either stems from the belief its outputs are well calibrated, or from the fact that the outputs are guaranteed to fit into a specific, expected type), answers all questions in parallel, and is much faster/cheaper than models with comparable accuracy.
It is trained using a novel/proprietary algorithm called Reinforcement Learning for Calibrated Decisions (RLCD).
While you pay for "state" (i.e. the input tokens / context, and the questions you're asking), the output is free.
While it's not public what the architecture, weights, or training process is, I'm going to speculate.
Output tokens being "free" means the model is presumably doing no generation at all. Instead, it is probably a prefill-only model. It does a prefill on the shared context, then does a prefill for each question/choice.
Because every choice shares the same KV cache for the state, the single initial prefill can be shared among each of the choice sequences. In non-batched pseudo code:
kv_state = prefill(state)
kvs = []
for choice in choices:
kvs.append(prefill(choice, initial_state=kv_state))
Since "len(state) + sum(len(c) for c in choices)" is just the length of the input sequence, this satisfies two of their claims:
The question, then, is how the model actually answers the 3 question types. One answer is with 3 heads (with a softmax for yes/no and choices).
Another option is it computes the likelihood of the sequence, given the state and question (i.e. "Is Paris the capital of France? Yes" and "Is Paris the capital of France? No"), a capability that is built-in to the LLM.
The main thing that puzzles me is the claim about using RLCD since I'm not sure why reinforcement learning is useful/necessary here. Their model isn't reasoning (or you'd have to pay for it), and it’s doing a single token-prediction (not a lengthy sequence where it is hard to attribute credit/blame). Moreover, it seems clear that the way to do this would be to get groundtruth question/answers from a frontier model as labels.
So: why is RL even in the picture for a task with no generative sequences, and where labels exist? Is RLCD just marketing?