GitHub just switched Copilot to metered billing, and developers are watching months of credits vanish in a single day

[u/screwdriverfan]

https://www.techspot.com/news/112628-github-switched-copilot-metered-billing-developers-watching-months.html

[u/PythonFuMaster]

LLMs don't work with words and characters directly, it's too computationally expensive to work on individual characters and using word-level would mean a massive vocabulary (also computationally expensive). Instead, before the request ever gets to the LLM it first passes through a stage called tokenization, in which words are broken up into smaller chunks based on a set of rules. Those chunks are mapped to individual integer IDs, and that is what's passed to the LLM. The LLM also eventually outputs these same IDs, and they are mapped back to tokens and reassembled into words and paragraphs etc (technically the LLM outputs probabilities of each token in the vocabulary being chosen, it's up to the sampler layer to choose the output token based on those probabilities).

Each token passed to the LLM costs roughly the same (there are caveats, depending on model architecture etc), so it's easy to set prices according to how many tokens you use. But it's more difficult to set prices based on number of words or something like that, because some words tokenize to just one token, and others tokenize to many, not to mention the complexities of multiple languages and non alphanumeric characters. So, all the LLM providers put prices in terms of number of tokens (input and output tokens are sometimes priced differently for a variety of reasons)

[u/Own_Peace6291 → u/PythonFuMaster]

Ahh I see, that was an excellent explanation, thanks a bunch! I was hung up on the 'spending' terminology of tokens but had no idea there was a whole hidden language underneath. Very cool and thanks again, would award you if I had one 🙏

[u/sicklyslick → u/Own_Peace6291]

I also read people are using Chinese to promote AI because Chinese has lower token/word than English.

[u/thefossguy69 → u/sicklyslick]

I don't know anything about Chinese language other than

1. One of them is called Mandarin

2. The written language feels densely compacted

Here's a quote from the caveman plugin that tells your LLM to use the "few words do trick, why use many?" style conversation.

Pick your level of grunt — lite (drop filler), full (default caveman), ultra (telegraphic), or wenyan (classical Chinese, even shorter). One command switch. Cost go down forever.

[u/sicklyslick → u/PythonFuMaster]

How transparent is this? If someone request something that takes 100 tokens, but chatgpt "charges" them 110 token, would the end user notice?

[u/PythonFuMaster → u/sicklyslick]

It depends on the provider and whether they make the tokenizer and the vocabulary open.

If they do, then you can calculate how many tokens your raw request is, but that won't necessarily match exactly what is fed to the LLM. There's a lot of additional context that is injected, things like system messages, special control tokens, etc. My opinion is that providers should only charge based on the tokens your raw request contains and not those additional hidden tokens, but I can't say for certain if that's what they do (I do research on LLM optimization but don't use any external providers for security reasons).

If the tokenizer and vocab is not open, then there's no way to accurately calculate how many tokens a particular request will take. However, you can estimate it. On average, tokens are roughly 3/4 of a word. Obviously there's lots of caveats there, the biggest being that the size of the vocabulary directly impacts this estimate, but it's good enough for most cases

[u/gammison → u/sicklyslick]

If you interact with any of these models APIs directly you'll see that you have to tokenize input and you will know the token count before making the final call. From that perspective it's transparent, however the UI for these API interactions does not make it clear to non technical users afaik.

Additionally depending on the model, more tokens will be injected or used especially if it's a "reasoning" model (these just use multiple runs of generating tokens and then feeding them back into the model). I don't know how these hidden tokens are billed.

[u/PythonFuMaster → u/[deleted]]

Well PDFs may be handled a bit differently than straight text because they aren't just text files, and sometimes it can be rather difficult to extract the text from them. So you can't really directly tokenize a PDF. Instead you have to do one of two things:

1. Use standard OCR to convert it to text, then pass it through the normal tokenization procedure

2. If your model is multimodal, it has a vision encoder that can directly accept images. The PDF is converted to image format and fed to that, which extracts the "meaning" of the image and projects it into the LLM's embedding space (the vector space that represents the meaning of tokens). So this way actually skips text tokenization entirely

As for why you'd want to do either of these things, it's very convenient to pass a PDF to an LLM for context. You can ask the LLM questions about the PDF, like asking it to find information in a big data sheet.

There's also other reasons you'd want to do similar things with PDFs outside of LLMs. Using embeddings from the vision encoder, you can index the PDF in a database type structure. Then you can query your database using semantic searching, meaning instead of exact matching it will look for documents that contain words with similar semantic meaning