Skip to main content

Let’s say there are 1000 records in Airtable that I need to fetch via the API on a regular schedule (to sync to an SQL DB). How is one supposed to keep within the API limits?



Since Airtable seems limited to only fetching 100 records at a time, this will need 10 requests, and I would assume that fetching them in quick succession could potentially hit the API limit of no more than 5 requests per second.



Can anyone confirm the best way around this? Is it necessary to introduce deliberate delays between the requests in order to combat this?



(The limit of only 100 records at a time is quite frustrating and bothersome IMHO.)



Simon.

The node.js library has a built in retry logic.



However if you are using some other native language like JAVA, PHP, .NET etc to directly interact with the Airtable API via their REST API you might want to have a global counter to keep track of the number of hits on the API per second.



Cheers!

FYI, here is how I rate limit using their provided API. I use library called Bottleneck. Ironic that their supplied node code doesn’t have built-in rate limiting.



// We create a few functions for updating/creating Airtables.


// We wrap them with limiter to make them adhere to 5 requests per second.


// Other requests that are not wrapped will not be subject to the limiter.


// I was finding a lot of rejected requests before I limits the APIs.


require('dotenv').config()


var Bottleneck = require("bottleneck")


var base = new Airtable({apiKey: process.env.ATAPIKEY}).base('apprICzWatICCBWdu')


const limiter = new Bottleneck({minTime: 1000/5}) // 5 requests per second


const limitedAirTableSiteCreate = limiter.wrap(base('Sites').create)


const limitedAirTableSiteUpdate = limiter.wrap(base('Sites').update)


const limitedAirTableTrafficCreate = limiter.wrap(base('Traffic').create)


const limitedAirTableTrafficUpdate = limiter.wrap(base('Traffic').update)

FYI, here is how I rate limit using their provided API. I use library called Bottleneck. Ironic that their supplied node code doesn’t have built-in rate limiting.



// We create a few functions for updating/creating Airtables.


// We wrap them with limiter to make them adhere to 5 requests per second.


// Other requests that are not wrapped will not be subject to the limiter.


// I was finding a lot of rejected requests before I limits the APIs.


require('dotenv').config()


var Bottleneck = require("bottleneck")


var base = new Airtable({apiKey: process.env.ATAPIKEY}).base('apprICzWatICCBWdu')


const limiter = new Bottleneck({minTime: 1000/5}) // 5 requests per second


const limitedAirTableSiteCreate = limiter.wrap(base('Sites').create)


const limitedAirTableSiteUpdate = limiter.wrap(base('Sites').update)


const limitedAirTableTrafficCreate = limiter.wrap(base('Traffic').create)


const limitedAirTableTrafficUpdate = limiter.wrap(base('Traffic').update)

nice - that is helpful Patrick.



fyi - you may want to scrub out your base id just to be safe.

Reply


Terms & Conditions