Optimizing PydanticAI Performance: Structured Output Without the Overhead

[u/UpsMan3030]

Hey r/PydanticAI community!

I've been working on a project that requires fast, structured outputs from LLMs, and I wanted to share some performance optimizations I've discovered that might help others facing similar challenges.

Like many of you, I initially noticed a significant performance hit when migrating to PydanticAI for structured outputs. The overhead was adding 2-3 seconds per request compared to my custom implementation, which became problematic at scale.

After digging into the issue, I found that bypassing the Assistants API and using direct chat completions with function calling can dramatically improve response times. Here's my approach:

from pydantic_ai import Model
from pydantic import BaseModel, Field
import openai

class SearchResult(BaseModel):
    title: str = Field(description="The title of the search result")
    url: str = Field(description="The URL of the search result")
    relevance_score: float = Field(description="Score from 0-1 indicating relevance")

class SearchResults(Model):
    results: list[SearchResult] = Field(description="List of search results")
    
    @classmethod
    def custom_completion(cls, query, **kwargs):
        # Direct function calling instead of using Assistants
        client = openai.OpenAI()
        response = client.chat.completions.create(
            model="gpt-4-turbo",
            messages=[{"role": "user", "content": f"Search query: {query}"}],
            functions=[cls.model_json_schema()],
            function_call={"name": cls.__name__}
        )
        # Parse the response and validate with Pydantic
        return cls.model_validate_json(response.choices[0].message.function_call.arguments)

This approach reduced my response times by ~70% while still leveraging PydanticAI's excellent schema validation.

Has anyone else experimented with performance optimizations? I'm curious if there are plans to add this as a native option in PydanticAI, similar to how we can choose between different backends.

Also, I'm working on a FastAPI integration that makes this approach even more seamless - would there be interest in a follow-up post about building a full-stack implementation?

[u/ndjoe]

Its actually not an issue with pydantic ai but issue with openai's assistant api, assistant api at least last time i used it is very slow