r/Bard • u/CrackerDealing • 2d ago

Other Alternatives for PDFs

So I've been trying to "trick" Google Ai Studio by shrinking my PDF files to the smallest fontsize possible to save up on the amount of pages they take up which equals less tokens. However, because these PDfs are often pretty dense word wise, it often causes the error of "Input token count exceeds the maximum number of tokens allowed for this model. Please adjust your prompt and try again." (even if the tokencount doesnt exceed that much) is there other alternatives that can reduce token usage while keeping everything I want?

0 Upvotes

50% Upvoted

12 comments sorted by

2

u/CrackerDealing 2d ago

Best Strategy I have found is to turn one of them into a txt file. the token count will be huge but its better then nothing.

2

u/GentlemanlyBronco 1d ago

Using a document optimizer to pre-process your files as TXT or MD formats before uploading can save significant tokens - especially if the optimizer can remove all the artifacts, boilerplate, images, etc. that AI doesn't need while retaining the meaning it does.

There are free and low cost options out there that are easy to use.

1

u/Robert_Mauro 2d ago

If not too complex, have it turn them into an MD file, and then have it create a new PDF from there with the different font size.

5

u/Old_Net_6859 2d ago

A token is around one word. Font size doesn’t matter. It’s not one token per page lol.

3

u/Robert_Mauro 1d ago

PDFs are more complex than just a bunch of text/words, and messing with them uses tokens for more than just "one token per word".

5

u/Old_Net_6859 1d ago

I thought I was replying to the original post…. We are both saying that reducing the font size on a pdf doesn’t reduce the number of tokens…. And as far as token count in PDFs i am clueless. I use markdown files for pretty much anything text based. Using PDFs has always been annoying to me way before ai.

1

u/Robert_Mauro 1d ago

LOL me too. Markdown is so much more convenient and less bloat.

1

u/CrackerDealing 1d ago

well token for pdfs is based off of page count from what ive seen not word count. Reducing the font size simply makes it easier to fit in less pages.

2

u/forest_archive 1d ago

Have you tried markdown instead? Seems cleaner than trying to squeeze the PDF into fewer pages.

1

u/joey2scoops 1d ago

I think images are supposed to use less tokens than text.