I’m trying to sanely bulk update a set of records from a source to Airtable. Mechanically, I have it working, but I’m thinking that I need to employ some better practices for managing these updates.
Here’s the current state:
Have a JSON payload of multiple rows, preprocessed to match the Airtable schema
Do not have the Airtable record ID in that dataset, we have our own identifier
Loop over each record in the JSON payload, add a timestamp (for sanity checks)
Run a select query on the API filtering by our ID
If nothing found, issue a create request with the payload
If found, issue an update request, passing the record ID from Airtable returned in 2
As written a number of async requests get issued and we run into conflicts. Then data doesn’t get written. I want to make sure the sync is bullet proof so we don’t run into data loss issues or have to do a lot of additional QA. I mitigated by setting a timeout for the first request. I’m thinking of two ways to improve this:
Separate out the select and just bulk match the records in a pre-processing step, cache that (we’re only talking 1000ish records)
Use promises or async await to wait until a record gets updated or created before moving to the next record in the payload. The tradeoff is this turns it into a sequential process, but I think async is less important than data fidelity.
Do other folks have code samples or examples of doing a similar bulk process where you have a lot of updates on records? My main goal is to have confidence that data is loading to Airtable.
Additionally, as an aside, a bulk option for updates would be great so that we can just send the record payload or send it in chunks and get back a summary response when the operation completes.