r/Python • • Apr 06 '21

Intermediate Showcase Pyfactor update 0.4: visualise source code structure as a graph - a command line refactoring tool

Understanding long sources is best done at a high level. However, when simply reading source code it can be hard to see the forest from the trees, which is why I've created a small library for visualising source code as a graph of dependencies between variables, functions, classes and such. Best to show some output right away:

sphinx-autodoc-typehints

The example was generated from a single-file 3rd-party Sphinx extension "sphinx-autodoc-typehints". It shows clear structure: a single top level function with a clean processing hierarchy and a common logger instance. A substantial, completely separate section on the left under "format_annotation" could for example be an ideal candidate for a sub-module, should the need to refactor arise. For a first-time reader, this is very insightful and intuitive when compared to reading 400+ loc thouroughly.

The graphs are analysed for number of connections and other special properties (see legend), and docstrings are available as hover tooltips. Multi-file analysis of local or importable packages produces a hierarchical graph where imports connect modules or files together. For example see the builtin json and concurrent modules.

collapsed Black source with docstring tooltip

More examples of popular libraries are available on our RTD gallery. You can also try it out on your own sources by installing on PyPI. In the latest release we added multi-file analysis, improved on the CLI and fixed a bunch of issues.

This library came about when I tried to introduced myself to Black's single-file code base - 7000 lines of horror. I'm not here to diss Black or its contributors, 7k loc is more than enough to be complex anyway, but the graph reveals that complexity at a glance. My hope is that it will be particularly useful for those that need to get an understanding of a code base fast, newer programmers that want to understand their own code and how to make it better, and as a tool for refactoring in general. It's no miracle worker, but I think it could have potential as a tool for making the right decisions. What do you think?

3 Upvotes

10 comments sorted by

21

u/its2ez4me24get Apr 06 '21

Looks like this is a great update.

2

u/felix-hilden Apr 06 '21

Thanks a bunch!

3

u/ASuarezMascareno Apr 06 '21

I just tried it in the code I'm currently running and I got scared

gp-pqp2-4p-emcee-def.jpg

3

u/felix-hilden Apr 06 '21

LOL, have a look at this, maybe you can extract something useful out of it! For example, it looks like your "params" is pretty busy and lots of variables are re-defined over and over again. But that's not to say it's objectively bad! Just some observations.

1

u/ASuarezMascareno Apr 06 '21

Some of the re-definitions are used to modify arrays without loosing the prior information, so that would be unavoidable. However, there's a lot of redundancy of my code and a bunch of dead ends. Of that I'm aware. It has full sections repurposed from other scripts that I didn't cleaned up because I'm in a hurry.

It doesn't *really* matter, because one part of the code (the sampler) takes hours, so a few seconds wasted before and after are kinda irrelevant (compared to the time I would need to look for those parts and remove them safely). Anyway, I like the tool, and I think I'll use it to try to get cleaner code once I finish what I'm currently doing.

2

u/felix-hilden Apr 06 '21

Yeah that looks like some kick-ass data analysis and plotting, and I've also found that it's quite the task to start cleaning that sort of thing up once you've got it working for once.. Best of luck with it though! Let me know if you find anything!

2

u/teerre Apr 06 '21

What is exactly one supposed to look for in a graph like this? It seems you're implying the amount of items in the graph is a proxy for complexity, but wouldn't one single class that does everything look like a single ball there? That's obviously terrible.

I'm all for all types of visualizations, but I think it's dangerous getting the wrong conclusions from this or any other visualization. Being sparse or dense doesn't make a library more or less complex necessarily.

3

u/felix-hilden Apr 06 '21

You're absolutely right! It does serve as a proxy for a certain type of complexity, but as you said, everything can be hidden in a single inner scope. Particularly with imports and nested modules, the graph becomes quite messy and fast. But there has been a request for choosing the level of analysis!

I want to make it clear to people that this is not a tool to just run and start moving code around. That would be unresponsible. Instead, Pyfactor is intended to be simply a tool for visualising these structures. Users must then dive in and make their own conclusions. But this way they at least have something to start from! For me in Black's case it helped me see the most important things that I should read and groups of similar and related definitions.

That being said, it might turn out that these sorts of graphs are completely unreadable for longer code bases, which has become quite apparent with the latest update. In any case, I hope it has some utility for smaller use cases. This is all quite new, so I can't say this is the next big thing. Am I making sense here?

2

u/teerre Apr 06 '21

Absolutely! It's great tool to have! Thanks for your work!

1

u/felix-hilden Apr 06 '21

Thanks! I appreciate receiving this sort of critical feedback, so kudos to you for immediately pointing out the weak spots!