Remove iTunes Dupes with One Command on QNAP or MacOS
Published on 15 Mar 2009 at 5:01 pm.
No Comments.
Filed under Mac,QNAP,Tech.
Edit 2009-03-16: After using this to delete dupes on my QNAP I found while comparing the backup file to my library that there were a few (3 out of over 100) cases where somehow the original file did not exist. For example Kanye’s song “I Wonder” only existed in my library as “I Wonder 2.m4a” Again here it seems most of the problems are with music purchased from iTunes, I suggest not manually moving any purchased music. In any case the information here will work to remove dupes on your QNAP or Mac but a much better solution would be a script that checks to see if an original version of the file exists before deleting. I’ll be out of town for the next few days, but when I get back I’ll whip up a Perl script to do just that.
So I followed the instructions for moving my iTunes library to my QNAP 509. Against my better judgement I did check the box to allow iTunes to organize my music. I think in the long run it will be for the best, but in the short run what I got was a bunch of dupes. Mostly all my bought music that I had already copied over to the QNAP.
Looking closer and after seeing this on a buddy’s Mac once already, I found that all the duplicate files, even ones in their own directories ended in ” 1.mp3″ or ” 1.m4a”. So a quick find command, showed me all my dupes.
Edit 2009-02-16: If you do a library import more than once on the same shared directory the file names may end in 2.mp3 2.m4a or 3.mp3 etc instead of 1. You can alter your find command to include multiple numbers like this: find . -name “* [1-3].m*” -print However be very careful once you go past 1, because lots of songs end in ‘part 2’ ‘side 2’ etc.  I also added a post on removing the duplicates in iTunes after removing them from the file system.
find . -name “* 1.m*” -print
Now that I knew I could get at them I just had to brush up on my awk and xargs to finish the job. The reason I needed awk and xargs instead of just issuing -exec rm in the find command is because the filenames have spaces in them. The awk and xargs on the QNAP are very stripped down so it took a while to figure that one out but I finally got the job done with this command:
QNAP Find and Delete Dupes
Make sure you paste these commands all on one line!
Please note: Some systems don’t properly copy the double and single quotes when you copy these commands. I suggest you copy them into a plain text document first and replace all the single and double quotes before executing them in a shell. I tried pre tags and different fonts but the problem persisted.
find . -name “* 1.m*” -print | awk ‘{print “\42” $n “\42”}’ | grep -vi “Track 1” | xargs /bin/rm
I really wanted a -i on the end there because I do have some files called “Track 1.mp3” when they couldn’t get CDDB info. However the QNAP xargs doesn’t support the -o option so the -i was being ignored and nothing was happening. To get around that I added the grep -vi “Track 1” to preserve those files and let the /bin/rm run it’s course. Make sure you look at the output without the | xargs /bin/rm on the end before you add it back to delete. And always back up the files before you issue the command. An easy way to backup the files you will be delete is to just use the same find command and send it to tar.
find . -name “* 1.m*” -print | awk ‘{print “\42” $n “\42”}’ | xargs tar -cvf save_before_delete.tar
MacOS Find and Delete Dupes
For MacOS the xargs is better so the solution was much easier.
Get to your iTunes Music folder usually ~/Music/iTunes/iTunes Music
Backup the Files first:
find . -name “* 1.m*” -print | awk ‘{print “\42” $0 “\42”}’ | xargs tar -cvf save_before_delete.tar
Watch what it tars up and you can examine the tar file with:
tar -tvf save_before_delete.tar | more
Hit spacebar to view the list of files page by page.
If you don’t see any files in there that you don’t want to delete you are ready to move on. If there are files in there that you want to save, you can work out grep -v commands to exclude them, or you can just save them and put them back after the mass delete.
Ready for Delete:
find . -name “* 1.m*” -print | awk ‘{print “\42” $0 “\42”}’ | xargs -o /bin/rm -i
The -i on the end is going to make you confirm every file. So if you have a lot, you may want to look at the list and use grep -vi to remove files you don’t want to delete ahead of time, and then you can issue the command without the -i.
Once you have examined your library and made sure nothing was deleted that you didn’t intend you are free to delete the backup tar file.