How to fetch more than 50 records

[Mike_Pakulak]

Hi all!

I’m working on a script that grabs two fields from a table as query parameters in a API GET request, gets the resulting JSON data, and updates the records in the same table for those corresponding queries. I have my for loop running fine when I limit it to 50 records (notice the IF statement) and aware of the 50 record limitation by Airtable. How do I go about refining my script so that it grabs the remaining records? I’ve been banging my head against a wall on how to do this since Friday.

Here’s the script I have so far:

//Set table for Stage Data

let testTable = base.getTable("Test Table");

let stages = await testTable.selectRecordsAsync();

let recordsData = [];

//Testing for loop

for (let record of stages.records) {

    let url = `https://apiwebsite?filter[account][customId]=${record.getCellValue("Query Param")}&filter[stage][id]=${record.getCellValue("Query Param")}`

    

    if (recordsData.length < 50) {

    let data = await fetch(url, {

        method: 'GET',

        headers: {"Authorization": `Bearer ${temp}`}

    })

    const results = await data.json();

    let newRecord = { id: record.id, fields: {"Total Prospects": results.meta.count}};

    console.log('newRecord', newRecord, record.id);

    recordsData.push(newRecord);

}}

while (recordsData.length > 0) {

      await testTable.updateRecordsAsync(recordsData.slice(0, 50));

      recordsData = recordsData.slice(50);

}

[kuovonne]

Are you using Scripting app to call the Airtable REST API?

Scripting App and the REST API have different limits.

The 50 record limit is for Scripting App to make batch changes (create, update, or delete records). The selectRecordsAsync function in scripting does not have a record limit unless you specify specific options.recordIds. If you use options.recordIds, then you are limited to 100 record ids.

If you are reading records (from a different base) using the REST API, there is a maximum of 100 records returned per page. If you want more than 100 records, you need to follow the instructions in the REST API documentation for requesting multiple pages.

If you are creating, updating, or deleting records using the REST API, the batch limit is 10, not 50.

---

In Summary …

REST API

• read records: max 100 records returned per page request
• create, update, delete records: max 10 per request

Scripting App

• read (no record ids specified): no record limit
• read with record ids specified: max 100 record ids
• create, update, delete records: max 50 per request

[Mike_Pakulak]

Hi kuovonne!

I’m calling an external API to get the data I require, parsing the JSON and putting that information into a single column that corresponds with the query params. So in this case, the fetch limit is 50, as that was the error I kept getting after it would hit 50 requests.

Should further clarify as well. My end goal is an automation that gets triggered every morning, runs the script, grabs the API data, and updates the records. In order for this to work I need to work around the 50 API fetch limit for an external API.

[Mike_Pakulak · Accepted solution]

So I think I’m starting to see the limitation of script invocations. So if I have this right, the Airtable Script Invocation only allows 50 fetch requests. So if I have more than 50 fetch requests, I would have to separate it into a new script.

Edit: So I have it working now, with separate scripts since my API GET requests are in batches that are close to 50. So for those looking to do something similar, the hard limit PER SCRIPT is 50 fetch requests to an external API.