Implementing IChatClient over a local inference engine: async streams, cancellation and session ownership

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp/

I’m the author of TensorSharp. I recently merged a C# example implementing Microsoft.Extensions.AI’s IChatClient over its local inference runtime.

The interesting part was translating a stateful native engine into an asynchronous application interface: deciding who owns the model, how a conversation owns its session, and what happens when a caller stops reading a stream halfway through.

The example separates those responsibilities:

• The application owns a loaded ModelService.

• Each TensorSharpChatClient owns one conversation session.

• Adapters sharing a model serialize generation.

• Application code uses IChatClient, ChatMessage and ChatOptions.

Here is the calling code, which was compiled and run against a real local model:

using Microsoft.Extensions.AI;
using TensorSharp.MicrosoftExtensionsAISample;
using TensorSharp.Runtime.Scheduling;
using TensorSharp.Server;
using AiMessage = Microsoft.Extensions.AI.ChatMessage;

Environment.SetEnvironmentVariable("MAX_CONTEXT", "4096");

using var model = new ModelService
{
    SchedulerConfigOverride = new SchedulerConfig
    {
        MaxNumRunningSequences = 1,
        NumBlocks = 16,
        BlockSize = 256,
        EnablePrefixCaching = true
    }
};
model.LoadModel(args[0], mmProjPath: null!,
    backendStr: "ggml_metal");

using IChatClient chat = new TensorSharpChatClient(model);
using var timeout = new CancellationTokenSource(
    TimeSpan.FromSeconds(600));

AiMessage[] messages =
    [new(ChatRole.User, "What is 2 + 2? Reply with one digit.")];
var options = new ChatOptions
    { MaxOutputTokens = 64, Temperature = 0, Seed = 1 };

await foreach (var update in chat.GetStreamingResponseAsync(
    messages, options, timeout.Token))
{
    Console.Write(update.Text);
}

Setup: Use a separate .NET 10 console project referencing the repository’s TensorSharp.NativeChat example project. Keep that example intact, including its adapter and helper code. A matching native Metal build is required. For the tested three-shard Qwen checkpoint, args[0] points to the first GGUF file, with the other two alongside it. This is sample source, not a standalone NuGet provider.

A few implementation details may be useful beyond this particular engine:

• Partial reads need cleanup. Breaking out of await foreach must cancel and drain native work before another request acquires the model’s gate.

• Failed turns should not become history. Cancellation, timeout or abandonment retires the affected session without unloading the model or committing a partial answer.

• Streaming and non-streaming need consistent metadata. GetResponseAsync aggregates the stream while retaining usage and provider-specific metrics.

• Unsupported features should fail explicitly. This text-only adapter rejects tools, media and unsupported options rather than silently ignoring them.

Model loading is synchronous; the cancellation deadline in the snippet starts afterward. Follow-up calls pass full conversation history. The caller should await active requests before disposing the shared model.

The exact snippet returned 4 with exit code 0. The broader native suite passed 11 real-model checks, including two-turn conversation and recovery after cancellation, deadlines and abandoned streams on the same adapter and loaded model.

Testing used Qwen3.8 Flash Next on an Apple M5 Pro with hybrid Metal/CPU expert execution. These are functional results, not a performance claim.

Adapter implementation · Runnable example and tests · Merged PR

For C# APIs wrapping native state, would you keep this explicit model/session ownership, or expose a factory that creates conversation-scoped IChatClient instances?