A while back, I posted some data deduplication routines and a document on Google Drive describing their use; you can find the applicable URLs in this reply, with a mention of an improved version a little lower in the thread. If you can wait a few days, I’m in the process of revising the document and sample bases, and I hope to have them online sometime next week.