Improve app performance on large bases/load tables conditionally

[Sticker_ninja]

Hello! I asked here a couple of weeks ago whether my apps are really slow because of the number of records in my table. It turns out that is the case, but I have a couple of followup questions.

1. Is the decrease in performance related to the number of records size of the base or of the individual tables?
2. Is it possible to load a table or base into an app conditionally? And if that is possible, would it even improve the performance of my apps? My experiments so far result in the “too many hooks rendered” error. For ex:

const base = useBase();
   let records=null;
   let table=null
    if(something is true)){
        table=base.getTableByNameIfExists('Line Items');
        records= useRecords(table)
    }

3. Any other suggestions for how to improve app performance on large bases/tables? I don’t think I really have the option of deleting records or archiving them in another base because my system of linked records makes it complicated.

Thanks so much!

[kuovonne]

You have a hook inside a condition. I find it works better if you put the hooks at the top level of the component so that it is always called the same number of times every time the component is rendered. Then, only conditionally render the component.

Also, if you only need limited data in response to a user clicking a button in your app, you can wait to load the data only when the user clicks the button.

[Fred_Zhao]

+1 to everything Kuovonne mentioned. In fact, regarding hooks inside conditions, React actually requires hooks to be used without conditionals. You can learn about it here, which also includes a helpful lint tool to ensure your code adheres to these requirements.

By the way, for a quick intro, I’m an engineer at Airtable working on the @airtable/blocks SDK, and I’m actively working on improving app performance on large bases. :slightly_smiling_face: So I’m glad you’re asking these questions.

I’ll answer your questions in a bit of a mixed order, which hopefully sets up the various answers more logically. :nerd_face: Let’s jump in!

For question 2: When a table is loaded, we actually don’t load the records by default, so the call base.getTableByNameIfExists(...) is actually very fast. So what’s the problem? It’s the call to useRecords(...) – that’s when we actually load the records, and where performance will become problematic.

For question 1: Yes, the decrease in performance is related to the number of records in the table passed into useRecords(). If there are lots of records, then this would result in loading a lot of data. (To a lesser degree, the performance also relates to the number of fields, which I’ll get into as part of question 3 too.)

Before answering question 3 of ways to improve performance (which I’ll post as a separate comment, as it’ll have a lot of details), I’ll take a brief interlude here.

I’m curious about your app’s use case, as I’d love to understand your end goal here! What problem is your app trying to solve? Does your app need all the records, or just some records (perhaps from a specific view)? Does it need all the fields, or just some specific fields?

Feel free to be as detailed or as generic as you need to be, if the use case is sensitive knowledge. Thanks in advance! Now I’ll start drafting a reply for question 3…

[Fred_Zhao]

For question 3, the answer has 2 parts: what you can do now, and what you can look forward to with some improvements we’re actively working on.

What you can do now

(Note: These steps can be tried out individually or together. They really depend on your end goal and what your app’s needs are, so that’s why I was curious and asked about them in my reply above.)

• [Mitigate] Delay loading the records. This is exactly the second part of Kuovonne’s comment above. Can your app wait to show the main UI (that is, the React component calling useRecords()) until the user clicks or interacts with the app in some other way? If so, then that helps shift the performance issue so it won’t happen immediately on render. Note that this doesn’t fix the issue, but does mitigate it.
• [Fix] Make use of fields. useRecords() has a second, optional argument called opts which is documented here. If your use case only needs specific fields, then specifying them in the fields key of this argument will allow Airtable to load less data. If the table has many fields, this could result in loading significantly less data!
• [Future-proofing fix] Use a filtered view. useRecords()'s first (required) argument can be a Table or a View. If your use case works just as well with a filtered view as it does with a table, then passing in that view as the first argument here will also allow Airtable to load less data. Note this won’t give you an improvement yet – see the next bullet point for details. But because switching your app from a table to a view changes your product’s user interface implications, you can get started on this right away.

What you can look forward to

• [Fix] Use a filtered view, along with a new future SDK version. Note that the 3rd bullet point in the last section is a work in progress. One of the improvements we’re making is a new SDK minor release version to fully leverage the fact that a view needs fewer records loaded. By “minor release version” I mean that you won’t need to rewrite any other part of your app, just bump the package.json dependency once it’s available. I’ll be sure to follow up here once that new version is ready!

[Sticker_ninja]

Hey thanks so much for your detailed answer! :slightly_smiling_face:

As for the use case, we have a printing business and we use several apps to track where orders are in the production process. There are different views corresponding to the different production stages.
In order to improve performance, I am hoping to be able archive orders no longer actively in production. However we sometimes reuse those “completed” records if the customer wants to reorder the same product, so I need to be able to unarchive them easily with their linked record relationships in tact. Since the number of records loaded into the app matters more than the size of the base as a whole, it seems like the best plan for now is to make another table on the same base for those completed orders (at least until that table becomes too large)

I will use the opts argument with useRecords() – thanks for for that tip! And I really look forward to the SDK update! :slightly_smiling_face:

[Sticker_ninja]

Thank for you reply! Do you mean I can call useRecords on a component that isn’t the parent component of the app and load that component conditionally?

[kuovonne]

Yes, you can call useRecords on a component that isn’t the parent component of the app. For example, in some of my apps, if the settings are not configured properly, I have a different component that tells the user to configure the settings instead of my main component that loads data. If the settings aren’t configured properly, there is no point in loading data.

Also, to clarify what I was saying earlier, not only can you delay calling useRecords until you really need it, you can also consider whether or not you actually need to use all those records. If you only need one or two records, only load those particular records. For example, you could try useRecordById() if you know the record you want from either the cursor or record action data. If you have one-off data processing, you could use selectRecords.

[Fred_Zhao]

Thanks for sharing the context, this is great!

As you’re accurately anticipating, separating records into a separate “completed orders” table just delays the problem down the road. It might also not be ideal because you would lose the history of a record such as edit history and comments in the original record’s table.

So before you create that separate table, try out opts with fields. This should help your performance right away, and I’d love to know if it’s enough to make everything better!

If opts with fields turns out not to be sufficient by itself, I think there’s another thing to try in order to avoid creating the separate table. This builds on the “future-proofing fix” solution above by leveraging views. To flesh out this idea a bit:

1. You would still keep using a single table for all the records, across all the different production stages
2. One view would filter out orders no longer actively in production
3. For the case of reusing “completed” records, we can create a view with a more complex filter, such as “(Status is not ‘Completed’) OR (Creation date >= ‘1 year ago’)”. The idea is that this view includes more records than point 2., but isn’t quite as many records as an entire, unfiltered table
4. (Optional) If this complex view still doesn’t turn up any search records to help reorder the same product, then maybe the app can finally fallback to searching the entire unfiltered table
5. Then, with the SDK update, hopefully you can bump the SDK version and see another performance improvement :slightly_smiling_face:

[Sticker_ninja]

Awesome, thanks again! :slightly_smiling_face:
Do you guys have an idea yet of when the SDK update will come out?

[Fred_Zhao]

No exact timeline yet while we’re stabilizing the package, but if you’d like to try a pre-release version just to see how much potential benefit it has, I can DM you a link! No warranties included for the pre-release, so I wouldn’t recommend keeping it in your production build. :winking_face: But still might be helpful to help you see whether it even makes a difference in performance.