The core idea

Deduplication

SSH tar streaming

Real benchmarks

Getting started I'm a systems engineer. I spend a lot of time copying files — backups to USB drives, transfers to NAS boxes, moving data between servers over SSH. And I kept running into the same frustrations: So I built fast-copy — a Python CLI that copies files at maximum sequential disk speed. When you run cp -r, files are read in directory order — which is essentially random on disk. Every file seek on an HDD costs 5-10ms. Multiply that by 60,000 files and you're spending minutes just on head movement. fast-copy does something different: it resolves the physical disk offset of every file before copying. On Linux it uses FIEMAP, on macOS fcntl, on Windows FSCTL. Then it sorts files by block position and reads them sequentially. That alone makes a big difference. But there's more. Many directories have duplicate files — node_modules across projects, cached downloads, backup copies. fast-copy hashes every file with xxHash-128 (or SHA-256 as fallback), copies each unique file once, and creates hard links for duplicates. In my test with 92K files, over half were duplicates — saving 379 MB and a lot of I/O time. It also keeps a SQLite database of hashes, so repeated copies to the same destination skip files that were already copied in previous runs. This is the part I'm most proud of. Instead of using SFTP (which has significant protocol overhead), fast-copy streams files as chunked ~100 MB tar batches over raw SSH channels. The remote side runs tar xf - and files land directly on disk — no temp files, no SFTP overhead. This even works on servers that have SFTP disabled, like some Synology NAS configurations. Three modes are supported: Local copy — 92K files to USB: Remote to local — 92K files over LAN: The simplest way — just run the Python script: Or download a standalone binary (no Python needed) from the Releases page — available for Linux, macOS, and Windows. For SSH transfers, install paramiko: I'd love to hear feedback — especially from anyone dealing with large file transfers or backup workflows. What tools are you currently using? What's missing from them? Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse