Help

This Product Ideas board is currently undergoing updates, but please continue to submit your ideas.

[SOLVED] Record Duplication Detection, Deduping, and Duplicate Merging

cancel
Showing results for 
Search instead for 
Did you mean: 
Aaron_Owen
7 - App Architect
7 - App Architect
Status: Launched

I would love to have a feature that would search a given table by a field or set of fields to find and surface duplicate rows. It would then be great to be able to take a pre-defined action on duplicated rows such as delete, or merge. Dealing with dupes is always a pain with large data sets or importing new data into a set where some records may already exist. A way to deal with dupes in a somewhat automated way would be really helpful.

93 Comments
Anna_Campbell
5 - Automation Enthusiast
5 - Automation Enthusiast

Occasionally in our video production database, we find the same project is represented by two different records – often both with partial information filled out in separate fields. It would be immensely helpful to have an option to merge two similar records into one, without having to manually copy and paste all relevant info.

William_Nutt
6 - Interface Innovator
6 - Interface Innovator

Important feature. Please implement!

Brendyn_Montgom
5 - Automation Enthusiast
5 - Automation Enthusiast

Seems like a logical addition would be able to have some kind of merge duplicates functionality especially when dealing with complex databases, it’s important to preserve linking between tables which makes manual deduplication a pain in the bum.

W_Vann_Hall
13 - Mars
13 - Mars

I recently created an updated version of my De-duplication Demo base that does away with the need to create a massive match field for each record. It seems to operate more crisply than the original.

Jeff_Hammerbach
4 - Data Explorer
4 - Data Explorer

I’m currently using a Google Sheets Add-on which works well. I’d love to use Airtable, but without this feature, I can’t move from Google Sheets.

W_Vann_Hall
13 - Mars
13 - Mars

Please take a look at my previous message. As long as you can create a link from each row of the table you want checked for duplicates to a single row in another table — a process that can be as simple as marking a range of cells and pasting a period into the entire range — you can have duplicate detection with Airtable today.

Grant_Dailey
6 - Interface Innovator
6 - Interface Innovator

+1 For having a native feature built into Airtable for detecting duplicate records.

I use Airtable for youth sports registration databases. When a parent registers their child, the information is sent to Airtable via Zapier. Often parents will register twice, sometimes three times at varying points in the registration period. For any interested, here is my workflow for finding duplicates:

  1. I setup my column on the far left to display “Percent Unique”

  2. If that number is less than 100, I group the records by the far left column, collapse all, then skim the columns looking for the groups that have more than one record.

I get that this is a bit clunky, but I am only working with <300 records.

David_Browne
5 - Automation Enthusiast
5 - Automation Enthusiast

Hi, just started using this and am running into issues with duplicates. It would be great to be able to look at a side by side comparison of a new record that gets added that has some of the same information as an old record, and easily choose what gets merged, and what information gets left behind.
For example, a potential client fills out a form that populates a row with phone number, email address, and name, but there is another row already in the sheet with similar name, same phone number, different email, I’d like to make the call on which information gets merged.

W_Vann_Hall
13 - Mars
13 - Mars

I think you’ve managed to encapsulate both the benefits of and the difficulties in having Airtable incorporate de-duplication as a core function. Aside from such basic, building-block functionality as, for instance, ensuring uniqueness of values for a table’s primary field, most users’ de-duplication needs are highly specific — even circumstance-specific, at times. (For instance, when merging various databases, each new database added may bring with it a different set of criteria for identifying duplication.)

The approaches I outlined earlier support user-tailored detection routines and the application of ‘overrides’ to indicate when detected duplicates were permissible — both of which I would consider a minimum capability of any de-duplication function. Personally, I would rather see Airtable implement features that could be incorporated into de-duplication algorithms, such as Soundex[1] matching or field-level forced uniqueness, than attempt to build an all-encompassing, omnibus de-duplication engine.


  1. Soundex assumes English words and pronunciations; obviously, Airtable would likely need to support comparable approaches for other language sets.
Mike_Ashelby
5 - Automation Enthusiast
5 - Automation Enthusiast

Seriously - this is a constant topic of conversation, and the lack of attention it’s being given is a real shame. So many situations require merging of data like this, and removing duplicates seems like an obvious feature for AirTable - and one I’m up against now myself!