Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run.
In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
Amusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
So to be clear, you can't prompt-inject these?
... How did they get into the training data?
They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.
The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.
GPT-2 used <|endoftext|>
Can the same mechanism be used to make sure it outputs syntactically correct code?
Yes, absolutely. The problem is integrating it into the inference engine. Not impossible, just _very hard_.
Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.
https://github.com/huggingface/transformers.js/blob/main/pac...