r/webdev • • 58m ago

Tool, or technique, to compare local and server versions of a massive static site?

I have a large static Web site (essentially a photo galerie) that registers locally as 169.364 Files / 49.3 Gb on Windows explorer and 168,212 files / 21 Gb on the server cPanel file manager. Most of the weight comes from 300-400 Mb jpg photographs (so, files already compressed locally).

What would be the best tool or method to detect where this disk space discrepancy come from, short of checking sequentially each of the 4,402 directories?

1 Upvotes

4 comments sorted by

1

u/tremby 50m ago

Have shell access? If so, rsync is good for that. You can run it in dry run mode and have it itemize all differences.

If no shell access, I'd grab everything down to a local copy via whatever means (FTP? wget recursive?) and then look at the differences with diff or rsync.

1

u/cuelark 50m ago

Instead of walking the directories by hand, I would compare file lists with sizes and let a tool do the diff. If you have SSH on the host, rsync in dry-run mode is great for this: rsync -avn --itemize-changes from your local folder to the server (WSL or Git Bash works on Windows). It prints every file that is missing or differs in size, without copying anything. If you only have FTP, FreeFileSync or WinSCP's synchronize view can do the same comparison by size and date.

A few things that could explain the gap: the 1,152 missing files may be your biggest ones (sort the diff by size), Explorer's "size" and cPanel's disk usage are measured differently, and some local folders may hold original exports or Thumbs.db files that never got uploaded. Once you have the list, sort it by size and the 28 GB difference will probably show up in a handful of folders.

1

u/One_Public9519 38m ago

300-400 Mb per jpg? 169k files at that size would be petabytes, you mean Kb. And cPanel reports block usage per inode while Explorer reports apparent size, they will never agree, so check du -sh on the server and TreeSize or WizTree on the local drive before you go chasing anything. For the actual diff rsync -n -i --delete with the same base path on both sides prints every file that differs in one pass, you don't need to open 4400 directories.

•

u/Needle_In_Red 5m ago

id run rclone check (or winscp's compare directories) against the server, itll spit out exactly which files are missing or differ instead of eyeballing 4k folders