How to find duplicate files BY CONTENT!!

From www.ReeltoReel.nl Wiki
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Today in IRC suseROCKS needed to find all duplicate files in a directory by their content, not by their file name, so we whipped up this fancy little 1 liner bash script to do the trick:

find . -type f -exec md5sum '{}' \; | sort | awk 'dup[$1]++{print $2}'

EDIT:

As Andreas suggested, using xargs instead of -exec is much faster, here is the updated command:

find . -type f -print0 | xargs -0 md5sum | sort | awk 'dup[$1]++{print $2}'