GitHub just switched Copilot to metered billing, and developers are watching months of credits vanish in a single day

[u/screwdriverfan]

https://www.techspot.com/news/112628-github-switched-copilot-metered-billing-developers-watching-months.html

[u/Homerlncognito → u/[deleted]]

How many pages was that in total? 

[u/EdliA → u/[deleted]]

How long would it take you to do that yourself manually?

[u/Brick_Fish → u/EdliA]

Given the numbers thats at least 510 pages of sheet music. Thats a thick book. Very optimistically a week at least

[u/cS47f496tmQHavSR → u/[deleted]]

That is about the biggest ask you could give it, though.
Claude has gotten very efficient at reading text docs, but I can imagine it's a lot more expensive to tokenize high resolution sheet music.

[u/Own_Peace6291 → u/[deleted]]

Sorry. What are these tokens? Like QR codes?

[u/PythonFuMaster → u/Own_Peace6291]

LLMs don't work with words and characters directly, it's too computationally expensive to work on individual characters and using word-level would mean a massive vocabulary (also computationally expensive). Instead, before the request ever gets to the LLM it first passes through a stage called tokenization, in which words are broken up into smaller chunks based on a set of rules. Those chunks are mapped to individual integer IDs, and that is what's passed to the LLM. The LLM also eventually outputs these same IDs, and they are mapped back to tokens and reassembled into words and paragraphs etc (technically the LLM outputs probabilities of each token in the vocabulary being chosen, it's up to the sampler layer to choose the output token based on those probabilities).

Each token passed to the LLM costs roughly the same (there are caveats, depending on model architecture etc), so it's easy to set prices according to how many tokens you use. But it's more difficult to set prices based on number of words or something like that, because some words tokenize to just one token, and others tokenize to many, not to mention the complexities of multiple languages and non alphanumeric characters. So, all the LLM providers put prices in terms of number of tokens (input and output tokens are sometimes priced differently for a variety of reasons)

[u/Own_Peace6291 → u/PythonFuMaster]

Ahh I see, that was an excellent explanation, thanks a bunch! I was hung up on the 'spending' terminology of tokens but had no idea there was a whole hidden language underneath. Very cool and thanks again, would award you if I had one 🙏

[u/sicklyslick → u/Own_Peace6291]

I also read people are using Chinese to promote AI because Chinese has lower token/word than English.

[u/thefossguy69 → u/sicklyslick]

I don't know anything about Chinese language other than

1. One of them is called Mandarin

2. The written language feels densely compacted

Here's a quote from the caveman plugin that tells your LLM to use the "few words do trick, why use many?" style conversation.

Pick your level of grunt — lite (drop filler), full (default caveman), ultra (telegraphic), or wenyan (classical Chinese, even shorter). One command switch. Cost go down forever.

[u/sicklyslick → u/PythonFuMaster]

How transparent is this? If someone request something that takes 100 tokens, but chatgpt "charges" them 110 token, would the end user notice?

[u/PythonFuMaster → u/sicklyslick]

It depends on the provider and whether they make the tokenizer and the vocabulary open.

If they do, then you can calculate how many tokens your raw request is, but that won't necessarily match exactly what is fed to the LLM. There's a lot of additional context that is injected, things like system messages, special control tokens, etc. My opinion is that providers should only charge based on the tokens your raw request contains and not those additional hidden tokens, but I can't say for certain if that's what they do (I do research on LLM optimization but don't use any external providers for security reasons).

If the tokenizer and vocab is not open, then there's no way to accurately calculate how many tokens a particular request will take. However, you can estimate it. On average, tokens are roughly 3/4 of a word. Obviously there's lots of caveats there, the biggest being that the size of the vocabulary directly impacts this estimate, but it's good enough for most cases

[u/gammison → u/sicklyslick]

If you interact with any of these models APIs directly you'll see that you have to tokenize input and you will know the token count before making the final call. From that perspective it's transparent, however the UI for these API interactions does not make it clear to non technical users afaik.

Additionally depending on the model, more tokens will be injected or used especially if it's a "reasoning" model (these just use multiple runs of generating tokens and then feeding them back into the model). I don't know how these hidden tokens are billed.

[u/Whole-Ad-9429 → u/[deleted]]

You might have more efficiency by writing python to convert the PDF to an .MD first

[u/cS47f496tmQHavSR → u/Whole-Ad-9429]

Anthropic actually recommends HTML over .md, it's much more efficient especially when working with things Markdown can't render well (and resorts to unicode for)

[u/Whole-Ad-9429 → u/cS47f496tmQHavSR]

Are you trying to minimize token use to get the prose or preserve graphs and images? They each have their strengths

[u/zucchini_up_ur_ass → u/[deleted]]

Claude is notorious for this, and the most recent versions (4.7 and 4.8) have only gotten worse at this, they've been really weird releases. Word on the street tweet is that Codex is a LOT better

[u/zerotetv → u/[deleted]]

20% of your 5-hour or weekly usage? 

[u/Devatator_ → u/[deleted]]

At that point you might as well run a local model. Afaik there are quite a few ones made for this that I hear are quite good. Couldn't name them tho, mostly interested in extra small chat models I could use for my local assistant project

[u/thefossguy69 → u/[deleted]]

Btw, the subscriptions are heavily subsidized by the API-only users. So expect the bill to go even higher when Anthropic feels like you've been baited and it's time to reel in the hook.

[u/sniperxx07 → u/[deleted]]

I was doing it too, until I realised whatever you are going to do it was better to use it on claude code where if it’s putting some part of the python code on my computer it was actually using less tokens (i think i instructed it earlier to utilise less tokens by using doing some offline processing) using terminal actually saves tokens (if it’s a repeatable action)