Thanks for these pointers! I’ve tried parallelizing these async calls to do ‘mega-batches’, but I found that it makes no difference in how Airtable processes them. For example, compare these two loops (for reference, mstart is the script start time, and create is a Map of records that need to be created — I use a Map to prevent double creations for records referenced multiple times):
for (let i = 0; i < create.size; i += batch) {
let start = new Date().getTime()
console.log(`create batch ${i / batch} start: ${(new Date().getTime() - mstart)/1000}`)
const values = Array.from(create.values()).slice(i, i + batch)
await table.createRecordsAsync(values)
console.log(`create batch ${i / batch} done: ${(new Date().getTime() - mstart)/1000} (${new Date().getTime() - start}ms)`)
}
This gives the following output:
create batch 0 start: 4.525
create batch 0 done: 6.16 (1635ms)
create batch 1 start: 6.16
create batch 1 done: 7.025 (865ms)
create batch 2 start: 7.04
create batch 2 done: 7.841 (801ms)
create batch 3 start: 7.841
create batch 3 done: 8.807 (966ms)
...
When I change that loop to one that does not await the create (a naive implementation of mega-batches):
for (let i = 0; i < create.size; i += batch) {
let start = new Date().getTime()
console.log(`create batch ${i / batch} start: ${(new Date().getTime() - mstart)/1000}`)
const values = Array.from(create.values()).slice(i, i + batch)
table.createRecordsAsync(values).then( () => {
console.log(`create batch ${i / batch} done: ${(new Date().getTime() - mstart)/1000} (${new Date().getTime() - start}ms)`)
})
}
You can see that although the creates are started almost simultaneously, the seem to be throttled internally to one at a time and they still take almost exactly the same amount of time to process per batch:
create batch 0 start: 3.722
create batch 1 start: 3.76
create batch 2 start: 3.837
create batch 3 start: 3.841
...
create batch 0 done: 4.84 (1118ms)
create batch 1 done: 6.364 (2604ms)
create batch 2 done: 7.378 (3541ms)
create batch 3 done: 8.347 (4506ms)
(note that the ms count is not a reflection of the batch processing duration anymore in this case, but the end time compared to the end time of the previous batch is).
Any thoughts, is there something I’m doing wrong here?
