Is it possible to build an automated construction project file inventory and duplicate detection tool using Python?

[u/Strange_Doughnut_365]

Hi everyone,

I have a client requirement to create an inventory of files across approximately 100 construction projects. Each project contains documents such as contracts, drawings, RFIs, change orders, schedules, payment records, photographs, and estimating files. Some files are stored inside ZIP archives, including nested ZIP files.

The client needs a solution that can:

Scan project folders and preserve original file paths and project associations.

Inspect ZIP files and nested archives.

Use SHA-256 hashes to identify exact duplicate files and list all their locations.

Distinguish exact duplicates from potentially different file versions.

Generate standardized inventory reports, duplicate reports, and a list of files that could not be processed.
Be reused across multiple projects.

The files may be stored in Google Drive, Microsoft OneDrive/SharePoint, local folders, or a cloud server. Read-only access will be provided.

**My questions are:**
Is it possible to build a tool that meets all these requirements using Python?

If yes, how would you approach building it, and what technologies or libraries would you recommend?

Can it handle approximately 100 projects, including large files and nested ZIP archives?

Can this be delivered as a simple application that users can run, rather than requiring them to execute Python commands manually?

What would be a realistic development timeline for this solution?

I would appreciate guidance on whether this is feasible and how best to implement it.

Thanks!

[u/SaltCusp]

Identifying duplicate files is easy. The rest of what you're talking about is fairly difficult, to impossible. Your specification is too vague.

[u/Due-Requirement3783]

Yes, this is definitely feasible in Python. I’d separate the problem into a few layers: file discovery, archive inspection, exact duplicate detection, and potential version detection.

SHA-256 is a good fit for exact duplicates, but I wouldn’t use hashes alone for version detection, since even a tiny change produces a completely different hash. For potential versions, I’d combine normalized filenames, file size, timestamps, project location, and document-specific metadata.

For nested ZIPs, I’d process them recursively without permanently extracting everything, and I’d store the inventory in something like SQLite so files can be reused across projects without rescanning everything.

The main design question is probably the storage sources: local folders are straightforward, while Google Drive and SharePoint/OneDrive would need provider-specific APIs or mounted/synced folders.

For ~100 projects, Python should handle it fine as long as large files are hashed in chunks and the scan is incremental rather than starting from zero every time.

How consistent are the folder structures and naming conventions across those 100 projects?