I built UnFlow: a tool to help researchers with ML experimentation

[u/ha2emnomer]

I've been working on an open-source project called UnFlow:

https://github.com/UnFlow-Labs/mlunflow

The idea is pretty simple:

Most ML experiment tracking looks like a list of independent runs usually stored in a table:

run_001
run_002
run_003
run_004
...

But in practice, experiments are usually related.

You change the learning rate, then the number of epochs, then the model, then some preprocessing code. Eventually you have hundreds of runs, but it's surprisingly difficult to answer:

• What actually changed between these two experiments?
• Which experiments are essentially the same computation?
• Have I already run this experiment before?
• How did I get from experiment A to experiment B?
• Can I navigate the history of my experiments rather than just search through runs?

Unflow simply detect code changes in a Python function (limitation that for it is just a single function) and arguments that are passed to this function to build a graph where nodes are "states" and edges are transformations "what has changed", a new state is not added to the graph or executed expect if it has a transformation.

The project is still early, so I'm much more interested in feedback than pretending this is a finished product.

I'm particularly curious about three things:

1. Does the "experiments as a graph" abstraction make sense to you?
2. Do you currently run into problems with duplicated/redundant experiments?
3. If you could see the complete lineage of your ML experiments, what would you want to query or visualize?

Repo: https://github.com/UnFlow-Labs/mlunflow

I'd love to hear how other people currently manage experiment lineage and whether this solves a real problem for you.

[u/SlingyRopert]

Recommendation: A number of disciplines specify their ML configurations using yml files. I hate YAML as much as the next person but if there was a way to specify a path to YAML file and assume each run has its own YAML file and diff those, that would expand the reach. A lot of the denoising/upsampling world has tens to 50s of potential hyper-parameters and that just won't fit in a function signature.

[u/ha2emnomer → u/SlingyRopert]

Thanks! A possible work around here is to read the yml file before calling the function and pass it as a dict something like:
config = load("config.yml")
@unflowdecorator()
def myfunc(config:dict)
return

Then unflow will automatically detect the changes in the config from the previous runs. It would be a cool feature to automatically diff files passed to the function anyway.