Receive more than 100 records

Stefan_Welsh · ‎May 17, 2018

I have written a little vue.js app. I need it to access more than 100 records in my table though. How do I do that?

Here is my code.var app = new Vue({
el: ‘#app’,
data: {
items: []
},
mounted: function(){
this.loadItems();
},
methods: {
loadItems: function(){

                    // Init variables
                    var self = this
                    var app_id = "XXXXXXXXXXXXXXXX";
                    var app_key = "XXXXXXXXXXXXXXX";
                    this.items = []

axios.get(
“https://api.airtable.com/v0/"+app_id+"/Donor Honor Roll?view=Grid%20view”,
{
headers: { Authorization: "Bearer "+app_key }
}
).then(function(response){
self.items = response.data.records
}).catch(function(error){
console.log(error)
})
}
}
})

Any help is appreciated.

Dev_Local · ‎Dec 05, 2019

ANSWER: After some experimenting, here is the Node.js / GET solution, that answers the above.

Do not set GET param ?maxRecords, otherwise there will be no param offset returned in the first response.

offset is not an integer, as would be expected (like start at 101), but rather is an Airtable row identifier (character string).

To loop, and get all records, this works:

function processRecords(records) {
    console.log('processRecords()');
	for (let record of records) {
		const airtable_id = record.id;
		const fields = record.fields;
		// do stuff with fields: field['<key>']
	}
}

function getParams(offset) {
    const params = {
        offset: offset
    };
    return params
}

const processTable = async () => {
    const url = `https://api.airtable.com/v0/` + AT_API_BASE + `/` + AT_TABLE_NAME;
    
	const headers = {
		'Authorization': "Bearer " + atApiKey,
        'Content-Type': "application/json"
    };
	
    let fetchCount = 0;
    try {
        let offset = 0;    // no initial offset, explicitely set to 0

        while (true) {
            const params = getParams(offset);
            console.log('fetchCount:', fetchCount);
            await axios.get(url, {params: params, headers: headers})
                .then((response) => {
					fetchCount += response.data.records.length;
                    offset = response.data.offset;
                    processRecords(response.data.records);

                    //console.log(response.data);
                    //console.log(response.status);
                    //console.log(response.statusText);
                    //console.log(response.headers);
                    //console.log(response.config);
                });
            if (typeof offset === 'undefined') {
                break;
            }
        }
    } catch (error) {
        console.error(error)
    }
    console.log('fetchCount:', fetchCount);  // print final # records fetched
};

If there are improvements upon this in any way, whether it is directly related to the Airtable API or with Node (code or structure, async/await), or any other issues, please share.

Output:

fetchCount: 0
fetchCount: 100
fetchCount: 200
fetchCount: 300
fetchCount: 400
fetchCount: 500
fetchCount: 600
fetchCount: 700
fetchCount: 800
fetchCount: 900
fetchCount: 1000
fetchCount: 1024

neiro_mendez · ‎Dec 20, 2019

Hi all, now airtable return in the respond with the first 100 records with an offset, just used it to the second call with an async function
let resp = [];
try {
     do{
    let params = {
            offset: ""
        };
    await axios.get(url, {params:params, headers:headers}).then((response) => {
        for (let item of response.data.records)
          resp.push(item);
        if(response.data.offset)
         params.offset = response.data.offset;
        else
         params.offset = "";
    });
    }while(params.offset !== "");
} catch (error) {
        console.error(error)
} 

//you will get all the rest of records.

Bill_French · ‎Dec 23, 2019

Indeed, it cannot be an integer mapping to sequential record items because Airtable supports Queries by View and as you know, views can contain all sorts of filtering, ergo, it requires a logical mapping of data, not a numerical mapping.

aaron_bell · ‎Feb 25, 2022

why is something like this so complicated…?

Joe_Webster · ‎Mar 01, 2022

With The official Airtable JavaScript library you can now use all() instead of having to manually paginate:

    const data = await base(tableName)
      .select({ view: viewName })
      .all();

github.com

Airtable/airtable.js/blob/fbb8fb38872a3c4eecc048136efe3ac68ce0308b/src/query.ts#L186

      
        
                        }
                    });
                };
            
            
    inner();
            }
            
            
/**
             * Fetches all pages of results asynchronously. May take a long time.
             */
            function all<TFields extends FieldSet>(
                this: Query<TFields>,
                done: RecordCollectionCallback<TFields>
            ) {
                if (!isFunction(done)) {
                    throw new Error('The first parameter to `all` must be a function');
                }
            
            
    const allRecords = [];
                this.eachPage(
                    (pageRecords, fetchNextPage) => {