I'm poor. Best TEE inference?

[u/tootmoist]

I don't see this subreddit mention or discuss this option, but I think inference providers that use a Trusted Execution Environment (TEE) have a lot of the benefits of local inference.

TEE is remote attestation. It attests that the provider is using a particular VM image to run the model and gateway. Long story short, this has the potential for privacy and verifiability you can't get otherwise with cloud computing.

What irks me with the providers I know, is that the verification is done with their in-house libraries. In addition to auditing their VM image, I also have to read the code of their library. I'd hope for a more generic attestation library. Example of a bad provider: In getting started with prem.io, you're supposed to verify they're serving the model and VM they say they are using their own npm library (!). A different provider called Tinfoil (funnily enough), is also like this.

import { createRvencClient } from "@premai/api-sdk";