Skip to main content

Is there a better way to do this?

  • May 5, 2022
  • 4 replies
  • 25 views
V
Forum|alt.badge.img+16

I want to fetch records from another base (for reasons) to output in the scripting app. My code works but I must be unable to find much documentation on how to paginate Airtable. Is there a better way to get all records?

let offset = '';
while (offset != "stop"){
    let response = await fetch(`https://api.airtable.com/v0/${remoteBaseID}/${remoteTableID}?api_key=${apiKey}&offset=${offset}`);
    let json = await response.json();
    if (json.offset !== undefined){ 
        offset = json.offset
        remoteRecords.push(json.records)
        }else{
            remoteRecords.push(json.records)
            break
        }
}

Open to suggestions.

4 replies

kuovonne
Forum|alt.badge.img+29
  • kuovonne
  • Brainy
  • 6009 replies
  • May 5, 2022

Pagination in the API is a slow process. It is documented with the rest of the API documentation.

You might need to add in logic to make sure that you do not exceed the rate limit of 5 requests per second, although simply awaiting response may be enough.

I recommend including a list of fields in the request so that you only get the fields you need, instead of all the fields.

You loop condition seems a little unusual. You are testing if the offset is equal to the string “stop”, but it will never have that value. It works but it seems strange to test for a condition that you know will never happen.

Your method of pushing records into the array also looks a little strange. You will end up with an array of arrays of records, rather than a straight array of records. Maybe this is what you want, but it is not what I would expect.

Also be aware that anyone with access to your base will have access to your api key.

V
Forum|alt.badge.img+16
  • Vivid-Squid
  • Author
  • Inspiring
  • 529 replies
  • May 5, 2022

Pagination in the API is a slow process. It is documented with the rest of the API documentation.

You might need to add in logic to make sure that you do not exceed the rate limit of 5 requests per second, although simply awaiting response may be enough.

I recommend including a list of fields in the request so that you only get the fields you need, instead of all the fields.

You loop condition seems a little unusual. You are testing if the offset is equal to the string “stop”, but it will never have that value. It works but it seems strange to test for a condition that you know will never happen.

Your method of pushing records into the array also looks a little strange. You will end up with an array of arrays of records, rather than a straight array of records. Maybe this is what you want, but it is not what I would expect.

Also be aware that anyone with access to your base will have access to your api key.


@kuovonne Thank you for the reply.

Yes, i made some odd choices, I have read up a little more and fixed them. I do want to limit the records instead of getting every record in the fetch.

Can I add

filterByFormula=REGEX_MATCH%28LOWER%28${remoteField}%29%2CLOWER%28${requestedRecord}%29%29

to the URL?

kuovonne
Forum|alt.badge.img+29
  • kuovonne
  • Brainy
  • 6009 replies
  • May 5, 2022

Since you are calling the REST API, yes, you can use filterByFormula.

Justin_Barrett
Forum|alt.badge.img+21
  • Justin_Barrett
  • Inspiring
  • 4647 replies
  • May 5, 2022

Pagination in the API is a slow process. It is documented with the rest of the API documentation.

You might need to add in logic to make sure that you do not exceed the rate limit of 5 requests per second, although simply awaiting response may be enough.

I recommend including a list of fields in the request so that you only get the fields you need, instead of all the fields.

You loop condition seems a little unusual. You are testing if the offset is equal to the string “stop”, but it will never have that value. It works but it seems strange to test for a condition that you know will never happen.

Your method of pushing records into the array also looks a little strange. You will end up with an array of arrays of records, rather than a straight array of records. Maybe this is what you want, but it is not what I would expect.

Also be aware that anyone with access to your base will have access to your api key.


The delay from awaiting the response has been sufficient from my experience. A project that I worked on last fall involved retrieving hundreds of records from multiple other bases via the REST API, and it never exceeded the rate limit.


Terms & ConditionsAccessibility statement