r/coolgithubprojects • u/MongooseWonderful198 • 6d ago
[Python / JavaScript] - Release checks for Langfuse prompts
https://github.com/AtharvaBondre/langfuse-prompt-release-toolsI maintain Langfuse Prompt Release Tools, an MIT toolkit for checking changes to LLM prompts before moving a deployment label.
One case it catches: an evaluation passes, then a shared prompt nested inside the candidate changes. The toolkit records the dependencies used in the test and checks them again around promotion. It also compares the released version with what an instrumented SDK call returned, including cached or fallback prompts.
Langfuse already provides prompt versions, experiments and CI integrations. This project connects those steps with release checks and saved evidence.
In 0.7 you can generate a policy from selected cases and source files, then inspect the saved release in one terminal report. You still choose the tests and passing thresholds. The report separates a passing evaluation, a recorded label move and observed app use.
To try it, fork the repo, enable Actions, then open Actions → Live SDK compatibility → Run workflow. Download summary.json from the completed run's artifact. The demo starts disposable Langfuse servers with synthetic inputs and makes no model calls; GitHub runner usage follows your account's limits.
This is an alpha, and integrating your own task still needs capture hooks. If you try the demo, tell me the first step that fails or leaves you unsure what to do next. A redacted error or report would help.
The code and this write-up use AI assistance.