We have built an automation that syncs data from our CRM with Airtable. Each insert/update/delete in the CRM post a message to a webhook created from an Automation. This works perfectly fine for single transactions in the CRM but for a bulk update or an import in the CRM of e.g. 500 records it only processes a limited number of records (e.g. 168 out of 800).
Is there a limitation in the number of instances of a script that can run in parallel? And if so, is there a way in Airtable to work around this as I have no idea to buffer the webhooks coming in.
I’m not sure if there is a limit to how many times a webhook can be triggered per minute/second. You may want to email email@example.com about that, and post back here with what they say.
However, there are 2 other limitations:
Each webhook request has a maximum payload size limit of 100kb per request. So if you’re sending 800 records at a time to your webhook, it’s possible that you’re going over the 100kb limit.
Your billing plan for your workspace will give you a limited amount of automation runs that you can execute per month. Each triggered webhook counts as one automation run.
If you’re continuing to have problems with Airtable’s webhooks, I would highly recommend using the professional webhooks from Make, because they have no limits at all. No size limits, no run limits, and no limits on the number of times that you can trigger a webhook. They also offer custom webhook responses and custom mailhooks, too.
One webhook will not include 500 changed records from the CRM. Pretty sure this is not what’s happening.
So, I could run a trillion webhooks and there would be no phone call or consequences from Make?
Importing a big change into the CRM typically impacts any outbound webhooks one record at a time. But, this is likely to be very fast and will overload the available webworkers on the Airtable webhook receiver which I think is limited to five simultaneous workers at a time (I suspect records in excess are simply ignored by Airtable). I have a hunch the limit is identical to the API limit (5 per second I think).
The challenge is to throttle the CRM at times like this by ensuring that bulk injections include a status field of some type that either the CRM uses to avoid firing the web hooks or such field is used on the Airtable side to ignore such bulk entries. Then, use a more gated process to introduce these new uploaded entries into Airtable in a distributed fashion over time such as when that status flag is lifted, allow it to be posted to Airtable.
Adam here from the Airtable Support Content Ops team. Great question on the limit. Currently, the limit is 5 webhook requests per second along with the other 100kb size limit that Scott mentioned. I’ve gone ahead and updated our documentation to reflect that in the FAQs section. Hope this helps and thanks for tagging me in @ScottWorld
I had some conversations with Airtable staff on the issue and they explained:
@ScottWorld My (limited) understanding of Make is that it also uses webhooks to access Airtable. Wouldn't those webhooks also count against the limit of 10 per base?
I'm just starting to build out some automation scripts and Make scenarios, and I'm weighing this whole webhook thing carefully. Make isn't going to help if I need 20 scenarios that use webhooks.
No, you can create an unlimited number of webhooks and/or mailhooks in Make and use them with Airtable.
Airtable's limit only counts when you are using their own internal webhooks, and they are also extremely difficult to use, so it's best to use Make, where everything is extremely simple & straightforward.
I created a brief navigation video for Make here. And I also provide the links to a few other Make training resources there as well (some of which cover webhooks in more depth). For example, to instantly trigger your webhook scenarios from Airtable with a script instead of a button, check out this thread.
p.s. If you have a budget for your project and you’d like to hire an expert Airtable consultant to help you with any of this, please feel free to contact me through my website: Airtable consulting — ScottWorld