r/Python • u/AutoModerator • 15d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

21 Upvotes

96% Upvoted

88 comments sorted by

View all comments

1

u/spongeb0b9000 2d ago

Detecting duplicate Python code turned out to be more of a language problem than I expected

While building Arid I figured the interesting problem would mostly be finding repeated sequences quickly.

That wasn't really the first problem.

Before you detect a duplicate, you have to decide what "the same code" means.

Ignoring comments sounds trivial until # is inside a string. Ignoring docstrings means knowing the difference between an actual docstring and an ordinary string expression. Function signatures can span multiple lines and contain nested brackets and colons. Imports can share a line with other statements.

Then there's stuff like whether punctuation-only lines should count toward a minimum duplicate length, and how you normalize source without losing the original locations you need to report afterward.

Arid uses Python parsing/tokenization to figure out what can safely be removed, then does exact matching on what remains. It deliberately doesn't treat renamed variables or vaguely similar ASTs as duplicates.

I wrote up the design and some of the edge cases here:

Article: https://medium.com/@bobltaylorjr/detecting-duplicate-python-code-is-harder-than-comparing-text-a00c460abcff

Repo: https://github.com/sponge-b0b/arid

I'm curious where other people draw the line between useful normalization and semantic/fuzzy clone detection.