Millions of records airtable

[Michael_Riley]

Hi everyone - I saw @Bill.French post about handling millions of records on airtable, but still am unsure given my use case.

I’m building an inventory management app on airtable and building the front end on softr.io.

I’m needing to store 1+ million records (serial numbers). The workflow is like this: someone on softr searches the serial number → softr returns fields associated with the record

I.think something like this could work:

1. User enters serial number → creates a row on airtable
2. Once the row is created it triggers an automation to search for that serial number in some outside database that handles more records
3. Airtable pulls back the other info related to the record on the large database into the airtable row that was just created
4. Visualizes this data in Softr
5. Once done record is deleted

Does anyone have any thoughts? Thanks so much in advance.

[Bill_French]

Hi Michael, and welcome to the community!

Michael_Riley:

someone on softr searches the serial number → softr returns fields associated with the record

Ahhh… the search engine. Have you considered an inverted search index? It’s been my experience that the moment you place Airtable in between the user and the data, things slow down a lot.

Michael_Riley:

it triggers an automation to search for that serial number in some outside database that handles more records

This can certainly work, and latency wouldn’t be an issue if the external database were fast enough to return the results. In practice, this is a challenge and why I lean on an inverted index which can return results in milliseconds even on a very large data set.

Here’s a paper I wrote about this topic some time ago.