Best way to sync data with API

[chrisharper]

Hi,

I need to sync an Airtable base with another system via API. Currently, I am using Make/Integromat to run the HTTP request and then iterate through the responses to check Airtable to see if a record exists and then either update it if it does exist, or add a new record if it doesn't already exist. I run this daily. There are about 5,000 records currently, so the automation runs in Make add up each day. 

Lately, I have been doing more with the native Airtable automations using javascript, and I have an idea to try to use an Airtable automation to sync the data, essentially cutting out the Make scenario. It seems to me that reducing tools would streamline things and make them simpler. 

My question for you is if you think it is feasible for me to run a daily automation in Airtable that would send an HTTP GET request to the remote system, which will return a big JSON response with all of the remote records and their fields. The automation would then need to updated existing data in Airtable or add new data if a record is new. 

I'm just checking here to see if anyone knows of any pitfalls I need to be cautious of before I launch into this. Or, is it a bad idea to rely on an Airtable automation for this?

Thanks!

[ScottWorld]

You might already be doing this, but just an FYI that Make has an Upsert module for Airtable, which automatically creates the Airtable record if it doesn’t exist, or updates the Airtable record if it already exists. So all of that functionality is built into that one module.

[chrisharper]

Yes, I am using that module. However, the ID for each row of my data is not the Airtable record ID, it is something else. So I'm currently running an Airtable Search Records module first and then running the Upsert module. Is there some way to combine these that you are aware of? I couldn't think of another way to give the Upsert module the Airtable record ID. Or, is it possible to run the Upsert module using another unique identifier for a record?

[ScottWorld]

Nope, the way you're doing it -- with the Search module -- is the perfect way to do it! 🙂

[TheTimeSavingCo]

The biggest one would be the 30 second time limit for script actions I reckon