DataFetch is not run as an automation, so it does not have the same limitations.

There are various ways of splitting the script into batches. Most methods will need to have a field in the table that indicates when the update was last performed. Then the script will read that field to determine which records to include in the current batch.

You can split the processing across multiple scripting actions in the same automation (25 actions per automation at 50 fetches per script = 1250 records/automation), as well across multiple automations.

By the way, your script also is updating records one-at-a-time with updateRecordAsync. You will get better performance if you update records in batches of 50 using updateRecordsAsync.

Another option is to have a human remember to press an update button that runs a Scripting App at 8am every day. Scripting app does not have the same limitations as automation scripts.

Unless you need to update all fields in the target records, you should scope the query to read only the fields required for the process. This will improve performance just a little bit, but it could be significant if the table is loaded with fields unneeded for this process.

@kuovonne is spot on - this process is likely to be 50 times faster if the code were designed in a manner that compresses the number of writes into the target table.

This is another big one - you are making an API fetch for each record which is seriously impacting performance. Consider creating a query that extracts all rental data in a single fetch, and then perform the iterative process against the complete result set to update your Airtable records.

Both the code editor and the documentation currently discourage querying for records without specifying the fields. It is currently supported, but is likely to be depreciated eventually.

So, you should indicate the fields you need, even if you really need all fields in the table.

I hadn’t thought of this but it makes sense. I’m new to JS and building scripts on AirTable so there was a lot of trial and error to get to this point. I gave it a shot below…

So just to confirm, something like this:

let table = base.getTable("Ads");


let view = table.getView('On Market')


let query = await view.selectRecordsAsync({


    fields:  


        'Listing ID',


        'Status',


        'Street Number',


        'Street Name',


        'Unit',


        'City',


        'Rent',


        'Source']


});





let listings = await fetch(`https://www.domain.com/api/rentals/search.php?key=API_KEY`);


let listings = await listings.json();





for (let record of query.records) {





    for (let listing of listings) {





        if (record.listing_id == listing.id) {





            UPDATE RECORD





        }


    }


}

Thanks you @kuovonne

For the time being, I have implemented your last suggestion (human clicking a button in the scripting app) until I figure out how to properly implement some of the suggestions above in an automation.

let table = base.getTable("Ads");


let view = table.getView('On Market')


let query = await view.selectRecordsAsync({


    fields: f


        'Listing ID',


        'Status',


        'Street Number',


        'Street Name',


        'Unit',


        'City',


        'Rent',


        'Source']


});





let listings = await fetch(`https://www.domain.com/api/rentals/search.php?key=API_KEY`);


let listings = await listings.json();





for (let record of query.records) {





    for (let listing of listings) {





        if (record.listing_id == listing.id) {





            UPDATE RECORD





        }


    }


}

The placement of the UPDATE RECORD placeholder still suggests that you’ll be updating records one at a time. To follow through on one of Kuovonne’s suggestions, a common method of batch updating would be to

• declare let updates = t] (an empty array soon to be filled with record values to update) outside of your loops



• inside your if () statement do updates.push({id: record.id, {'Field Name': listing.field}})
  
  



• include something like this after and outside your loops to execute the record updates in batches of 50 at a time:

while (updates.length > 0) {


    await table.updateRecordsAsync(updates.slice(0, 50));


    updates = updates.slice(50);


}

let table = base.getTable("Ads");


let view = table.getView('On Market')


let query = await view.selectRecordsAsync({


    fields: f


        'Listing ID',


        'Status',


        'Street Number',


        'Street Name',


        'Unit',


        'City',


        'Rent',


        'Source']


});





let listings = await fetch(`https://www.domain.com/api/rentals/search.php?key=API_KEY`);


let listings = await listings.json();





for (let record of query.records) {





    for (let listing of listings) {





        if (record.listing_id == listing.id) {





            UPDATE RECORD





        }


    }


}

Yes, this will eliminate n(fetches) where n is the number of records in the table.

BUT!!!

There’s another seemingly innocuous optimization opportunity - nested loops.

What if - instead of looping through all listings looking for the ID of each table record, you used a more direct approach to that listing - a hash index that allowed you to point to the listing in 2 milliseconds.

If you created a hash index of the listings, you could lookup each listing instantly with a single reference. The for...listing loop is performed once to create the index, and all references to it are like this:

 if (oListings(record.listing_id])


   UPDATE RECORD

This eliminates a vast number of executions.

let table = base.getTable("Ads");


let view = table.getView('On Market')


let query = await view.selectRecordsAsync({


    fields: f


        'Listing ID',


        'Status',


        'Street Number',


        'Street Name',


        'Unit',


        'City',


        'Rent',


        'Source']


});





let listings = await fetch(`https://www.domain.com/api/rentals/search.php?key=API_KEY`);


let listings = await listings.json();





for (let record of query.records) {





    for (let listing of listings) {





        if (record.listing_id == listing.id) {





            UPDATE RECORD





        }


    }


}

Congratulations on what you have accomplished thus far in your coding journey. Simply getting a working script at all can be a major accomplishment.

You are getting great suggestions in this thread from some very experienced programmers. Some of the suggestions may feel a little overwhelming. That’s okay. Just see what you are capable of digesting in the moment, and work on that concept. If you find coding interesting, you might go through several versions of this script as you have time to figure out new things.