There are only two hard things in Computer Science: cache invalidation and naming things.
That said, the way I handle caching in Airpress is by creating a hash of the exact request sent to the API. Then I use a “refresh” and “expire” limit to deal with invalidation.
When the “refresh” limit (in seconds) is reached, the existing cached data is served while an asynchronous request is made to refresh the cached data. When the “expire” limit is reached the page-load must wait while new data is fetched.
To me, this was a reasonable approach for Wordpress sites that may have high traffic (and will utilize the refresh limit mostly) as well as low traffic sites that need caching but don’t want to serve old data.
Paging is the most difficult thing to deal with as Airtable will only return a maximum of 100 records per request. This is a problem because if you cache “page 4” of the results separately from “page 5” you’ll get duplicate (or missing) records when records are added / removed.
To get around this in Airtable I actually don’t do paging at all!! Gasp. I know. Instead, I’ll loop a request getting 100 at a time until all records are fetched and cache that result with the original request.
On the plus side, adding/deleting/updating records doesn’t mess with the cached records, but on the downside “showing all records” could be problematic from a memory, processing, or even storage standpoint.
However, most queries I’ve ever written against Airtable are filtered/limited to some sane number of records any way (by the way, I think 2000 records is perfectly sane, it’s only 20 requests!). And if that number starts getting high, then I do progressive requests where the first request is for the Record ID and Created At fields only, I then do subsequent requests for specific records (or batches of records) based on what the interface needs.
Beyond that, I would recommend creating an import process (as opposed to a caching process) that cycles through all your Airtable data at a given rate and imports (creates/updates/deletes) a “local” database and never have your application touch Airtable outside that sync/import. That way you’ll have your own “local” that can handle paging, caching, etc (and should be as performant as you want) while still being able to frequently update that data from Airtable. The limit here would be something like 30,000 records a minute ( 5 requests per second * 100 records per request * 60 seconds a minute).
What I do on other projects is create my import script and schedule it to run every 15 minutes. My longest (unoptimized) script takes about 5 minutes to run through several thousand records, fetching related records, updating the local DB, etc.
Hit me up if you want to talk further.
