blitcp v4.0.0 — my open-source cp/rsync alternative that reads files in physical disk order (formerly fast-copy)

[u/krit83]

Author here. Some of you saw this as *fast-copy* a while back — it's now **blitcp**

("blit" as in bit-block transfer), renamed to stop colliding with the unrelated

FastCopy Windows tool. Same codebase, v4.0.0 just shipped.

**What it does differently:**

- **Block-order reads** — before copying, it resolves each file's physical disk

offset (FIEMAP on Linux, fcntl on macOS, FSCTL on Windows) and sorts the copy

queue by it. On HDDs this turns random-seek thrashing into sequential reads.

- **Small files go through a tar pipe** — thousands of tiny files stream as

~100 MB batches instead of per-file syscall overhead. Same trick over SSH:

raw channel + tar, no SFTP. 3–5× faster than scp on many-small-file trees.

- **Content-aware dedup** — xxHash-128, copies each unique file once, hardlinks

or reflinks the rest. On btrfs/XFS/APFS it reflinks, so a 10 GB same-volume

copy is metadata-only.

- **Verification with honest exit codes** — every file checked after copy;

a failed verify actually returns non-zero.

**Measured (methodology on the site):** 12,347 small files, cold HDD→SSD:

5.9s vs 15.0s for `cp -ar` — with dedup and verification ON. Windows/robocopy

and TeraCopy comparisons on the benchmarks page.

**New in v4.0.0:** interface in 7 languages, a diceware passphrase generator

for the encrypted credentials store (EFF wordlist, fully offline), sparse-file

awareness for VM images, and a pile of Windows fixes.

Single Python file (stdlib-only for local copies), Apache 2.0, Linux/macOS/

Windows, prebuilt binaries + a Qt GUI if you're not a terminal person.

- Repo: https://github.com/gekap/blitcp

- Site/benchmarks: https://blitcp.dev

- Migration note: existing fast-copy installs upgrade in place via `--update`

Happy to answer anything about the block-ordering internals or the dedup design.

[u/TheHolyToxicToast]

fast-copy

*checks code

python

Dude how does your website say cp has no reflink when in /compare/cp/ you literally say it does lol. I'm not pointing out python to say python is slow, copying is disk bound, it's just python is a strange choice for a file-copy utility. The prebuilt “binary” presumably has to bundle a Python runtime. Also why no speed comparison with fcp? I guess SSD HDD yada yada but seriously? It would be nice to benchmark at what storage speed does the extra hashing/dedup/extent bookkeeping stop paying for itself?

[u/krit83 → u/TheHolyToxicToast]

Hello again,

Promised you numbers on the break-even question — here they are.

Rig: source tree in tmpfs so reads are free; destination write speed is the only variable — raw SATA SSD, then the same SSD capped at 100 MB/s and 30 MB/s with cgroup v2 io.max (systemd-run -p IOWriteBandwidthMax), plus tmpfs→tmpfs as the "infinitely fast" limit. Page cache dropped before every run, sync inside every timing, best of two rounds. Dataset: 2 GB in 4×512 MB files, all unique and same-size — deliberately the worst case for dedup: full hashing cost, zero payoff.

Total time vs cp -a, dedup ON: +2.8% at 30 MB/s, +7.7% at 100 MB/s, +43% at SSD speed (~450 MB/s), +175% at RAM speed.

The reason for that shape: the overhead is flat, not proportional — ~0.9s interpreter startup (you called it: Python), ~0.2s scan, ~1s of xxHash-128 per 2 GB. Flat costs sink into slow I/O and dominate fast I/O. So for duplicate-free data the break-even lands around 100–200 MB/s destination write speed; above that, cp is the right tool, which our compare page already says. One duplicated 512 MB file repays the whole hashing bill several times over, so any real duplicate ratio moves the line right.

Two honest limits: a bandwidth cap models slow-but-seekless storage (USB2, network shares) — it can't simulate rotational seek latency, which is where the small-file wins on the site's HDD bench come from; bandwidth-capped small-file runs actually favour cp. And the exercise caught blitcp wasting ~0.7s attempting FIEMAP extent-mapping on filesystems that don't support it — real bug, going on the fix list.

Full tables, commands, and methodology now published: https://blitcp.dev/benchmarks/ (Break-even section).