I would love to have a feature that would search a given table by a field or set of fields to find and surface duplicate rows. It would then be great to be able to take a pre-defined action on duplicated rows such as delete, or merge. Dealing with dupes is always a pain with large data sets or importing new data into a set where some records may already exist. A way to deal with dupes in a somewhat automated way would be really helpful.
Occasionally in our video production database, we find the same project is represented by two different records – often both with partial information filled out in separate fields. It would be immensely helpful to have an option to merge two similar records into one, without having to manually copy and paste all relevant info.
Seems like a logical addition would be able to have some kind of merge duplicates functionality especially when dealing with complex databases, it’s important to preserve linking between tables which makes manual deduplication a pain in the bum.
I recently created an updated version of my De-duplication Demo base that does away with the need to create a massive match field for each record. It seems to operate more crisply than the original.
Please take a look at my previous message. As long as you can create a link from each row of the table you want checked for duplicates to a single row in another table — a process that can be as simple as marking a range of cells and pasting a period into the entire range — you can have duplicate detection with Airtable today.
+1 For having a native feature built into Airtable for detecting duplicate records.
I use Airtable for youth sports registration databases. When a parent registers their child, the information is sent to Airtable via Zapier. Often parents will register twice, sometimes three times at varying points in the registration period. For any interested, here is my workflow for finding duplicates:
I setup my column on the far left to display “Percent Unique”
If that number is less than 100, I group the records by the far left column, collapse all, then skim the columns looking for the groups that have more than one record.
I get that this is a bit clunky, but I am only working with <300 records.
Hi, just started using this and am running into issues with duplicates. It would be great to be able to look at a side by side comparison of a new record that gets added that has some of the same information as an old record, and easily choose what gets merged, and what information gets left behind.
For example, a potential client fills out a form that populates a row with phone number, email address, and name, but there is another row already in the sheet with similar name, same phone number, different email, I’d like to make the call on which information gets merged.
I think you’ve managed to encapsulate both the benefits of and the difficulties in having Airtable incorporate de-duplication as a core function. Aside from such basic, building-block functionality as, for instance, ensuring uniqueness of values for a table’s primary field, most users’ de-duplication needs are highly specific — even circumstance-specific, at times. (For instance, when merging various databases, each new database added may bring with it a different set of criteria for identifying duplication.)
The approaches I outlined earlier support user-tailored detection routines and the application of ‘overrides’ to indicate when detected duplicates were permissible — both of which I would consider a minimum capability of any de-duplication function. Personally, I would rather see Airtable implement features that could be incorporated into de-duplication algorithms, such as Soundex[1] matching or field-level forced uniqueness, than attempt to build an all-encompassing, omnibus de-duplication engine.
Soundex assumes English words and pronunciations; obviously, Airtable would likely need to support comparable approaches for other language sets.
Seriously - this is a constant topic of conversation, and the lack of attention it’s being given is a real shame. So many situations require merging of data like this, and removing duplicates seems like an obvious feature for AirTable - and one I’m up against now myself!
