Download the scripts and tests (ZIP)
Python scripts, tests, and ready-to-run merge, join, and date practice files with expected results. No signup required.
Before cleaning a folder of downloaded reports, generate a list of files that appear to contain the same bytes. This script groups files by size, then compares their SHA-256 digests. It prints a JSON report and makes no changes to the scanned files.
Tested: Python 3.10 on Windows, September 24, 2026. A temporary test folder contained identical files, a different file of the same size, and two empty files. The test checked both the report and that every original file still contained the same bytes afterward.
A reproducible folder
Save this as make_sample.py in a scratch directory. It creates a new folder and stops if that folder already exists:
from pathlib import Path
folder = Path("duplicate-demo")
folder.mkdir()
for name, content in [
("a.txt", b"abc"),
("b.txt", b"abc"),
("c.txt", b"xyz"),
("empty1", b""),
("empty2", b""),
]:
(folder / name).write_bytes(content)
Run python make_sample.py. Both abc and xyz are three bytes long, so file size alone cannot tell you whether they are duplicates. Names cannot settle it either: two reports with different names may contain identical data.
The complete report script
Save this as duplicate_files.py next to the sample folder:
"""Report SHA-256 duplicate candidates; never delete or modify files."""
import hashlib
from collections import defaultdict
from pathlib import Path
def find_duplicates(folder):
root = Path(folder)
if not root.is_dir():
raise NotADirectoryError(root)
sizes, groups = defaultdict(list), defaultdict(list)
# Direct children only; explicitly skip symlinks.
for path in sorted(root.iterdir()):
if path.is_file() and not path.is_symlink():
sizes[path.stat().st_size].append(path)
for size, paths in sizes.items():
if len(paths) < 2:
continue
for path in paths:
digest = hashlib.sha256()
with path.open('rb') as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b''):
digest.update(chunk)
groups[(size, digest.hexdigest())].append(path.name)
return [names for names in groups.values() if len(names) > 1]
if __name__ == '__main__':
import argparse, json
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('folder')
args = parser.parse_args()
print(json.dumps(find_duplicates(args.folder), indent=2, ensure_ascii=False))
Run:
python duplicate_files.py duplicate-demo
Expected report:
[
["a.txt", "b.txt"],
["empty1", "empty2"]
]
c.txt is excluded because its contents differ from a.txt and b.txt. Empty files form their own candidate group. An empty report, [], means the scan found no matching groups under this script’s rules; it does not prove that an entire drive has no duplicates.
Verify the report before using it
Save this as verify_sample.py beside duplicate_files.py after creating the sample folder. Run python verify_sample.py with ordinary Python (without -O). This checks the actual command output and confirms that all five sample files still contain their original bytes:
import json
from pathlib import Path
import subprocess
import sys
folder = Path("duplicate-demo")
expected_files = {
"a.txt": b"abc", "b.txt": b"abc", "c.txt": b"xyz",
"empty1": b"", "empty2": b"",
}
def contents():
return {p.name: p.read_bytes() for p in folder.iterdir() if p.is_file()}
assert contents() == expected_files, "Start with the unchanged sample folder"
result = subprocess.run(
[sys.executable, "duplicate_files.py", str(folder)],
capture_output=True, text=True, check=False,
)
assert result.returncode == 0, result.stderr
assert result.stderr == "", result.stderr
assert json.loads(result.stdout) == [["a.txt", "b.txt"], ["empty1", "empty2"]]
assert contents() == expected_files, "Input files changed"
print("PASS: expected groups; all five input files unchanged")
Expected output is PASS: expected groups; all five input files unchanged. The checker uses subprocess.run to keep the report, error text, and process exit status separate. A successful scan exits with status 0 whether it finds groups or returns [].
Reproduce a missing folder and a scope limit
Run python duplicate_files.py duplicate-demo/missing while that path does not exist. Expect exit status 1, no JSON on standard output, and a traceback ending with NotADirectoryError and the missing path on standard error. Correct the folder argument to duplicate-demo and rerun; expect the original two groups. Do not treat missing output after an error as an empty report.
To see why a nested duplicate can be absent, save and run this as make_nested.py in the same scratch directory. It stops if nested-demo already exists:
from pathlib import Path
folder = Path("nested-demo")
folder.mkdir()
(folder / "child").mkdir()
(folder / "top.txt").write_bytes(b"abc")
(folder / "child" / "copy.txt").write_bytes(b"abc")
Now run python duplicate_files.py nested-demo. Expected output is [] and exit status 0 even though the two files have identical bytes. Only top.txt is directly inside the selected folder. The script uses Path.iterdir, which lists immediate children, and does not descend into child. Scanning the child separately also returns []; separate scans do not compare across folders. Choose a recursive inventory tool if your review must span a folder tree. This example does not provide that feature.
Why there are two passes
The first pass collects file sizes. A file whose size is unique cannot match another file byte for byte, so reading its contents would add no value for this report. The second pass hashes only size groups containing at least two files.
The hash loop reads chunks of up to one MiB. It does not load each full file into memory. The folder listing and result names are still held in memory, so this is not a bounded-memory indexer for millions of files. No performance benchmark is claimed here. The underlying digest interface is documented in Python hashlib.
Read the report as candidates
Matching size and SHA-256 is strong evidence of matching contents, but this script does not run a final byte-by-byte comparison and does not assess business importance. A copy in a backup folder may be intentional. Two paths can also be hard links to the same underlying file, so summing reported sizes would overstate recoverable disk space.
The report does not designate a file to keep. Review each group against your backup and retention needs before deciding what to do. This article supplies no deletion command. You can use the output as input to a separate review process without mixing identification and removal in one operation.
Scope and failure behavior
| Condition | Behavior |
|---|---|
| File is directly inside the chosen folder | Include it if it is a regular, non-symlink file |
| File is in a subfolder | Leave it outside this scan |
| Entry is a symbolic link | Skip it |
| Folder does not exist | Raise an error |
| File cannot be read or disappears during the scan | Stop with the original filesystem error |
The script deliberately avoids turning an unreadable file into a false “no duplicates” result. If it stops on a permission error, run it against a folder you can read and review which items were excluded. Do not change permissions across a drive simply to complete a scan.
Scan an unchanged local folder. If another application writes a file while it is being hashed, the digest might not describe a stable version. For shared or actively updated data, use a read-only snapshot and record when it was taken. A file can also be replaced between the symlink check and the open operation, so this example is intended for a trusted local workspace, not hostile directories.
For report data inside the files, follow the CSV merge tutorial. If your exports are line-delimited JSON, use the JSONL validator before importing them.
Code in this article is original Dadidan Lab example code. You may use and adapt it, including commercially. It is provided without warranty. Corrections: contact Dadidan Lab.
To check a transfer between two folders, create and compare SHA-256 file manifests. This preserves the distinction between duplicate content and changed files at matching paths.