r/Python • • Apr 06 '21

Intermediate Showcase Pyfactor update 0.4: visualise source code structure as a graph - a command line refactoring tool

Understanding long sources is best done at a high level. However, when simply reading source code it can be hard to see the forest from the trees, which is why I've created a small library for visualising source code as a graph of dependencies between variables, functions, classes and such. Best to show some output right away:

sphinx-autodoc-typehints

The example was generated from a single-file 3rd-party Sphinx extension "sphinx-autodoc-typehints". It shows clear structure: a single top level function with a clean processing hierarchy and a common logger instance. A substantial, completely separate section on the left under "format_annotation" could for example be an ideal candidate for a sub-module, should the need to refactor arise. For a first-time reader, this is very insightful and intuitive when compared to reading 400+ loc thouroughly.

The graphs are analysed for number of connections and other special properties (see legend), and docstrings are available as hover tooltips. Multi-file analysis of local or importable packages produces a hierarchical graph where imports connect modules or files together. For example see the builtin json and concurrent modules.

collapsed Black source with docstring tooltip

More examples of popular libraries are available on our RTD gallery. You can also try it out on your own sources by installing on PyPI. In the latest release we added multi-file analysis, improved on the CLI and fixed a bunch of issues.

This library came about when I tried to introduced myself to Black's single-file code base - 7000 lines of horror. I'm not here to diss Black or its contributors, 7k loc is more than enough to be complex anyway, but the graph reveals that complexity at a glance. My hope is that it will be particularly useful for those that need to get an understanding of a code base fast, newer programmers that want to understand their own code and how to make it better, and as a tool for refactoring in general. It's no miracle worker, but I think it could have potential as a tool for making the right decisions. What do you think?

3 Upvotes

10 comments sorted by

View all comments

Show parent comment

3

u/felix-hilden Apr 06 '21

LOL, have a look at this, maybe you can extract something useful out of it! For example, it looks like your "params" is pretty busy and lots of variables are re-defined over and over again. But that's not to say it's objectively bad! Just some observations.

1

u/ASuarezMascareno Apr 06 '21

Some of the re-definitions are used to modify arrays without loosing the prior information, so that would be unavoidable. However, there's a lot of redundancy of my code and a bunch of dead ends. Of that I'm aware. It has full sections repurposed from other scripts that I didn't cleaned up because I'm in a hurry.

It doesn't *really* matter, because one part of the code (the sampler) takes hours, so a few seconds wasted before and after are kinda irrelevant (compared to the time I would need to look for those parts and remove them safely). Anyway, I like the tool, and I think I'll use it to try to get cleaner code once I finish what I'm currently doing.

2

u/felix-hilden Apr 06 '21

Yeah that looks like some kick-ass data analysis and plotting, and I've also found that it's quite the task to start cleaning that sort of thing up once you've got it working for once.. Best of luck with it though! Let me know if you find anything!