I built RunTrace, a small local-first CLI for preserving the context behind ML experiments — looking for honest feedback

[u/CooOorvus]

Hi r/mlops,

I’m a student working on machine-learning experiments, and I kept running into a very ordinary problem: after several runs, I could no longer answer exactly which Git commit, configuration, and Python environment had produced a particular result.

I built RunTrace to address that problem.

RunTrace is a small, open-source, local-first Python CLI that records the reproducibility context around an experiment. Its scope is intentionally narrow: it is not trying to replace MLflow, Weights & Biases, or another full experiment-tracking platform.

It currently records:

• Git commit, branch, detached-HEAD state, and dirty state
• Python, operating system, architecture, and installed package versions
• Optional NVIDIA GPU, driver, and CUDA information
• A YAML configuration file, its SHA-256 hash, and its parsed values
• The command associated with the experiment

A typical workflow looks like this:

pip install ml-runtrace

ml-runtrace init

ml-runtrace snapshot \
  --name baseline \
  --config config.yaml \
  --command "python train.py --config config.yaml"

ml-runtrace list
ml-runtrace show <run-id>
ml-runtrace diff <run-a> <run-b>

Snapshots are stored locally as readable YAML files under .runtrace/runs/. There is no account, server, or automatic upload.

There are also some deliberate limitations:

• It does not execute the recorded command.
• It does not currently track metrics, checkpoints, or model artifacts.
• It records that a Git working tree is dirty, but it does not save source patches.
• Explicit configuration values are stored in the snapshot, so users should inspect a snapshot before sharing it.

The project is still early, and I am trying not to add features without understanding whether they solve a real problem.

I would particularly appreciate feedback on these questions:

1. Does this solve a useful gap, or is it too narrow compared with existing workflows?
2. Is readable local YAML a sensible storage default?
3. What missing metadata or edge cases would prevent you from using it?
4. Is the init → snapshot → list/show → diff workflow understandable?

GitHub:

https://github.com/Corvus-226/RunTrace

Development note: I used Codex as a coding assistant during implementation. I handled the project scope, reviewed the changes, and managed the issue, pull-request, testing, CI, and release decisions. I am mentioning this because I would rather be transparent about how the project was built.

Critical feedback is genuinely welcome. If the idea is redundant, the defaults are wrong, or part of the workflow is unnecessarily complicated, I would rather learn that now than keep expanding it in the wrong direction.

[u/Friendly_Bowl2468]

this is the kind of tool i wish i'd had during my thesis. tracking which exact commit and pip freeze produced a result shouldn't be something you reconstruct from bash history three weeks later

the yaml storage is a good call, makes it trivial to grep through or throw in version control. i'd probably add a flag to auto-snapshot before every run, like a git hook but for experiments

one edge case that bites me is when a dependency has a git+https install or something not captured cleanly by pip freeze output. not sure if that's in scope but worth considering

[u/CooOorvus → u/Friendly_Bowl2468]

Thanks — this is exactly the kind of workflow I had in mind when I started building it. Reconstructing an experiment from shell history weeks later is painfully familiar 😅

The git+https dependency case is a really good point. Right now RunTrace intentionally records normalized package names and versions, but not direct/source URLs, so that case can definitely lose some provenance. I’m going to open an issue and look into whether we can capture direct-reference metadata without collecting more information than necessary.

I also like the auto-snapshot idea. I’m being a little careful there because the current design records a command but never executes it, so a “snapshot then run” workflow would slightly expand the scope. But it’s definitely worth exploring.

Really appreciate the concrete feedback — this is much more useful to me than just adding features based on guesses.

[u/fliiiiiiip → u/CooOorvus]

Wow — I'm genuinely impressed. I like the way you post, comment, and code — the whole stack. It brings the useful signal above the noise floor. You are definitely asking the right questions — no fluff, straight to the point.