Discover and remove all duplicates cluttering your hard disk or mobile phone: Music and movies, Photos and pictures, Office documents, e-Books... or any other kind of clones!
Nowadays most people use Netflix or
similar streaming services for watching movies, but many old hands have
sizeable older movie collections, brilliant classics you won't find
streamed anywhere. No matter how you have amassed the movies, whether
from old DVDs or downloaded, chances are you have several copies of many
of those classic movies. If you think that the average movie has a file
size over 1GB, you don't need many of those duplicates to choke
up your hard disks. To tidy up your collection, save space and keep the
best quality version of each movie, you need a duplicate detector tool,
but how do you pick one? There are 102 alternatives to i-DeClone for Windows 7/10/11, both free and paid. This review will help you choose the best duplicate movie cleaner software.
Clearly the author is biased, but this review is objective. The most popular shareware programs that claim to deal with near-duplicate movie files are included, and their performance is measured
on a fixed set of video files (movie collection library). We believe
that the data are representative, almost scientific :) You are welcome
to try your own movie library for duplicates — you should find the same
results more or less, relatively speaking.
How do you discover near-duplicate video?
The technology behind duplicate file software is quite simple, almost
trivial. You just examine all the files in your hard disk byte-for-byte,
and discover all 100% identical files. This is fast and safe,
but will not work very well for your movie file collection. Video is
encoded compressed in various formats (along with its audio streams), in
containers such as Matroska MKV, the old Audio Video Interleave (AVI)
file, MPEG-4 (think MP3 but for video) and many more. Each container
allows multiple resolutions (720p, FullHD, UHD etc) for increased movie
quality (and byte size!).
So a movie that you or me would instantly recognize, is hard to tell
looking at the raw bytes stored in the file.
Recognizing movies calls for specialized software that can decompress
and extract the video and audio streams from the movie file, then use a
number of algorithms to compare them and discover similar movies. Some
programs compare a succession of thumbnails (frames) using similar image detection methods, others focus on the soundtrack and use audio fingerprints
to align and compare the audio streams. Such visual or audio algorithms
can be used to compare movies, and see through differences in video
resolutions, frame rates and dimensions, soundtrack loudness etc, to
identify visually identical files — as opposed to binary-identical.
Some simpler duplicate files software ignore the audio/video content and
try to find similar movies using file attributes like the name and
duration. This is next to impossible for movies, where there is no
universally accepted tagging system (like ID3 tags for MP3 music). Take
for instance the following 2 video files. It is the same movie, but the
file sizes and durations vary slightly, one is widescreen HD and the
other 720p — still they represent the same movie!
What we need therefore is software that can find movie content duplicates, allowing for differences in HD resolution and file formats (WMV,AVI,MOV,ASX,MKV,MP4,ASF,MTS,TTS...).
Media files need a computer program that can understand the video, not
the raw bytes stored in them. The bytes may differ, even the picture
quality itself slightly, but video-aware software will detect the
similarity, almost as a human movie watcher would.
Most of the aforementioned 102 duplicate cleaners are not "movie savvy". I have narrowed it down to just six (6) computer programs (for Windows 7/10/11) that analyze the movies for similarity and will find video duplicates, which in alphabetical order are:
Czkawka by Rafał Mikrut [CZK]
Duplicate Media Finder [DMF]
Duplicate video search by Bolide software [DVS]
i-DeClone by Zabkat [IDE]
Video comparer by Eric Bohain [VCO]
Video duplicate finder (open source, various contributors) [VDF]
I will also consider another popular program called Auslogics duplicate file finder,
just to show you how "naive" duplicate finders are not up to the task
of discovering duplicate movies. Some of these programs are free or open
source, and some shareware with free trial, so you would expect a decent quality in departments such as:
- Handle many video file types, not just AVI
- Easy to use for Average Joe consumer
- Affordable, preferably without annual subscriptions
- Discover many near-duplicates with the least effort (fast)
- Make few mistakes (false positives)
- Handle lots of files without bugs and crashes (robust)
Before we begin the Video Clone Olympics, here's a table summarizing the main contenders for the title of Best duplicate video detective
Windows software. Some are specialized for similar movies, others can
handle also similar music and images. 64-bit programs are capable of
handling huge media collections (32-bit programs are limited to 2GB
memory)
Information correct as of 18-Feb-2023
Typical workflow for finding and removing duplicate files
The workflow of all duplicate file removers more or less comprises 5 basic steps:
- Choose where to search. Define the search scope in certain folders or your entire PC and attached hard disks and other devices like phones.
- Choose what and how to search. We are searching for
movie files; there are many search options that affect the speed and
quality of the results. For this review we concentrate on similar video
content, and the most important parameter is the similarity tolerance,
a percent that shows how much similar 2 movie files should be to be
considered duplicate. Lowering the similarity threshold will catch more
duplicate movies but will also produce false positives, films that are
completely different.
- Wait for results and preview. Audio/Video fingerprint
scanning is slow by nature, but you don't need to sit on top of the
duplicate scanner while it is searching. Once the results are in, you
examine them to make sure the duplicate groups are correct. If you find
any mistakes (false positives) in the results, you can remove the wrong
files from further processing.
- Choose what to keep and what to remove. Once the search
is finished, duplicate files are grouped together and offer a way to
choose what is kept and what gets deleted. Mostly this is done using
checkboxes. Manual selection is impractical when many duplicates are
found, so programs offer various automatic ways to choose what gets deleted, e.g. the lowest resolution films, or the smallest file sizes (poor picture quality)
- Clean up marked items. Finally we want to remove
duplicates and free up the hard disk space. This is either done by
deleting the duplicate files, or replacing them with hard/symbolic
links, which are like file shortcuts that don't take up much space. Some
programs allow moving the duplicates in a separate folder, although I
don't see the point — other than perhaps as a temporary measure. The goal is freeing up disk space and reducing junk and clutter from our movie collection.
We are talking about similar movies, but what we are really after is identical
films. We are not trying to find Star Wars episode IV similar to
episode V, we want to see if we have Star Wars episode IV more than once
in our collection. "Similarity" enters the picture because no 2 movie
files are byte-identical, on account of resolution, quality and such
parameters.
|