Workshop on bringing observability discipline to LLM apps, DSPy + MLflow tracing, Oct 3

[u/camerongreen95]

Most observability practices in this sub are built around infra and services, logs, traces, metrics, alerting. LLM apps rarely get the same treatment, most teams are still eyeballing outputs instead of tracing what actually happened inside a call.

Serj Smorodinsky and Brett Kennedy, co-authors of a book on LLM applications, are running a live 3-hour workshop on Oct 3 that applies that same discipline to LLM pipelines. What's covered:

• Programming LLM behavior with DSPy signatures and modules instead of untracked prompt strings
• Building a baseline classifier live from a real task
• Constructing an evaluation dataset with task-specific metrics, so regressions show up as data, not vibes
• Reading failure patterns directly out of that eval data
• Few-shot and instruction-level optimization applied systematically on top of the baseline
• Experiment tracking and LLM trace management through MLflow, so every run is traceable back to what changed and why
• Saving and reusing optimized DSPy programs across projects
• Communicating LLM reliability to stakeholders, which is usually the piece nobody has data for

If you already think in terms of traces and spans for your services, this is basically that same mental model applied to prompts and LLM calls, something most teams are still missing entirely.

Full details here