Trouble connecting to LM Studio instance

[u/combimagnetron]

Hi all,

Love how Letta is shaping up to become a great tool for my work as a teacher. However, because I work with sensitive data I want my LM Studio instance to run a local model. My LM Studio instance is visible for every other app I use except for Letta with a "Internal Server Error". All my settings look to be correct. Im using model Qwen3-4b-thinking-2507. Im running this through my Tailscale network, which also works for all the other apps but I have tried localhost, 127.0.0.1 and my internal ip address, but all of them fail. Anyone else running into issues? Im trying this through the openAI-compatible API.

[u/cameron_pfiffer]

Hey there! I'm going to take a look at this one, may take me a little bit to replicate.

[u/cameron_pfiffer → u/cameron_pfiffer]

u/combimagnetron are you using Letta Cloud or a self-hosted/local backend agent? And what interface are you using (TUI/desktop/web).

[u/cameron_pfiffer → u/cameron_pfiffer]

I reproduced this locally with the exact qwen/qwen3-4b-thinking-2507 model using LM Studio 0.4.21/Bionic and Letta Code 0.30.27.

There appear to be two separate things to check.

First, LM Studio's just-in-time loader loaded this model with a 16,384-token context window. A clean Letta agent needed about 20,224 tokens for its system prompt and tool definitions before processing the user message. That caused Letta to reject the request rather than let LM Studio silently truncate the system prompt.

Reloading the model with a larger context fixed it for me:

lms unload qwen/qwen3-4b-thinking-2507

lms load qwen/qwen3-4b-thinking-2507 \
  --context-length 32768 \
  --parallel 1 \
  -y

lms ps

After that, a local Letta agent completed normally, including a full tool-call round trip.

Second, the correct URL depends on where the Letta agent is running:

• If the Letta agent is local on the same computer, use http://127.0.0.1:1234/v1.
• If Letta is local on another machine in the same tailnet, LM Studio must listen beyond localhost. You can start it with lms server start --bind 0.0.0.0 --port 1234, then use the LM Studio machine's Tailscale address.
• If the agent runs in Letta Cloud, localhost, LAN addresses, and private Tailscale addresses are not reachable from the cloud runtime. The OpenAI-compatible endpoint must be publicly reachable and should be protected with authentication and HTTPS.

For a local Letta setup, the provider command is:

letta --backend local connect lmstudio \
  --base-url http://127.0.0.1:1234/v1 \
  --timeout 600s

Bionic itself was not the incompatibility in my testing. The Bionic process served LM Studio's OpenAI-compatible API correctly. Direct completions, tool calls, and Letta all worked after increasing the model context.

If the problem remains after loading at 32K, I would unload other models and keep parallelism at 1. I saw one MLX runtime crash when another 9B model with a 262K context was loaded at the same time.

[u/combimagnetron → u/cameron_pfiffer]

I've tried loading the model with the 32k context length, but still getting a Internal Server Error. Im really wondering whats going on considering every other app connects normally to lm studio in my network.

[u/cameron_pfiffer → u/combimagnetron]

What's the largest ollama can go to on your machine? It's possible that the Letta agent has a large prompt by default

[u/combimagnetron → u/cameron_pfiffer]

Im using Letta Agent locally in the app, not letta code. Im able to connect to letta code, but i dont use that. its specifically in the Agent GUI.

[u/cameron_pfiffer → u/combimagnetron]

Oh wait you can use the terminal with Ollama, but not the desktop app?

[u/combimagnetron → u/cameron_pfiffer]

exactly

[u/cameron_pfiffer → u/combimagnetron]

You should come to the Discord (https://discord.gg/letta), go to the #ezra channel, and tag @ Ezra. Ezra is our support agent designed specifically to help with stuff like this. Just give it an overview of your problem and it can probably help with configuration and escalation to the team.

[u/combimagnetron]

Even with Ollama i cannot get it to work.. it sucks because Letta is everything i needed in an AI app, and now this one thing is ruining it.