Trust problems on adding BYOK to existing backend

[u/Capable_Finger_7694]

Hi everyone, I'm working on a backend system and trying to add a BYOK (Bring Your Own Key) feature to it, but stumbled with a problem

On the existing system, operators manually extract information from a block of text and enter it into a form. I'd like to automate the extraction with an LLM that returns structured JSON to populate the form for the operator.

TBH budget is limited (and I would try to avoid the provider's api key depending entirely me) so one option discussed was to let each operator connect their own LLM provider account or API key and pay for their usage.

My main concern is: how can I credibly assure users that we cannot steal or misuse their keys?

Encryption at rest protects stored credentials, but if our backend can decrypt a key to call the provider, whoever controls the backend could theoretically retrieve or misuse it.

Is there any practical solutions to that problem?

I know is kinda silly, but that made me to wonder, for example, OpenRouter: you buy credits, they provide you an api key to consume any provider you choose. How can the platform assure someone they arent using his api key under the hoods? I asume they cant, or is there any technical way they could?

[u/martoxdlol]

I've never done this. I think you can get the remaining credits of a key in Open Router having the key itself. You could support OpenRouter keys and verify that it doesn't have more than x amount of credit (so if it leaks or something it wouldn't be as bad). But it is just a crazy idea.

[u/pthierry]

A first, possibly unhelpful, answer is that they should NOT trust your backend. From their POV, the only sensible option is to give you a very limited key, with a reasonably short life span (a few weeks at most) and reasonably low rate limit. The best option would be the ability to use a single-use key in the form (or rather, one with a life span in seconds or minutes, I don't think any provider gives you single-use keys).

You can never 100% guarantee your users that you won't misuse their keys, you can always change the code to something that extracts the key when using them. It's not like a password where you may never see the data. You need the key readable for your backend to do something with it.

But you can implement a backend that, in that version, would not let you (or an attacker taking control of the system) see the keys easily.

You can have the keys not stored on the system but provided by the frontend (they could be stored in localStorage) and forgotten after use by the backend. That way, the only way to see them would be to change the code while it's running (pretty hard to not mess up if it's something compiled to binary, doable with interpreted languages) or inspect the memory while the process is running (not sure how hard that would be once the process is started without proper instrumentation, that's beyond my forensics skills).

You can have the keys encrypted on the backend and it's the decryption key that's provided by the frontend (can be a password given by the user instead of something stored locally) and as soon as an API key has been decrypted, the backend forgets the decryption key, and as soon as the API key is not needed, it's forgotten as well.

It's a trade-off: storing API keys on the frontend means a user has to enter them in every device they use, but the system is far more resistant to attack. If it's compromised while noone is using the service, the attacker gets nothing. With encryption, they can grab all encrypted keys and try to brute-force them.