Scalable migration?

Just a few days ago, Airtable increased the number of records in their enterprise plan to 250,000 records per base, which is a great step in the right direction from them. (Their pricing webpage hasn’t been updated to reflect this official change yet.)

With 250,000 records per base, perhaps you could create some sort of a record archiving system in Airtable to stay within each base’s limits?

But if 250,000 records per base still won’t be enough, I always highly recommend that everyone who outgrows Airtable to migrate to Claris FileMaker Pro (soon to be rebranded as Claris Pro), which is enterprise-level database software that has almost no limits at all. You can have unlimited records, unlimited automations, unlimited integrations, etc.

I was a FileMaker developer for 30 years. I’ve retired from FileMaker development to become a professional Airtable consultant, but if you reach out to me through my website, I can turn you onto some excellent FileMaker developers who can help you with your migration:

I have been asked to render opinions concerning the situation you find yourself in today and in almost every case (11 of them so far), the 50,000 record limit was determined without deep consideration of the context. In 9 of 11 cases, they had only one table that was near 50,000 rows and they assumed there was nothing they could do except leave Airtable. In 7 of those 9 cases, they remained on Airtable because they eventually realized that:

1. Their data model wasn’t well-optimized.
2. They had failed to identify the true nature of operational vs strategic data.
3. They could not envision an archive strategy that would continue to sustain their reporting objectives.

Are you possibly like the seven I have encountered?

Being up against the record limit is contextual. In my experience, the context that triggers a wholesale replacement of a perfectly competent platform is often misunderstood or not exhaustively researched before jumping ship.

Formula fields in Airtable alone are a unique beast that doesn’t translate well to any other platform.

I also have some experience with SeaTable and it is quite powerful in some ways and lacking in many because it doesn’t yet have the customer base needed to refine the platform. Changing from Airtable to just about everything will be unpleasant.

I’m guessing that you used at least three different techniques across those seven clients. I can guess at two possible techniques

• collapse selective data into json in long text fields, and only expand it when needed
• store data outside of Airtable and fetch it only when needed

I think you’ve written up about those techniques before, but I don’t recall the exact posts. I’d also be curious about any other tricks you use.

That’s quite the complement to Airtable.

Ha ha - not intended to be a compliment. Precisely the opposite actually. Airtable lulls complacency when it comes to database and app design. The formula field is a good example; most database systems do not have such fields of calculation type. SeaTable is at least one exception. But we tend to do a lot of things in Airtable to make stuff work that we would not do in other systems.

By “unpleasant” I’m really referring to the massive business logic that ends up in tables and fields, whereas, other competitors often have a different vision for implementing business logic. And they only need to be just a little different to make the transition very unpleasant.

They’re not tricks - just good modelling can often optimize storage and with a well-designed archiving approach, you can remain 100% in Airtable. Yes, it’s a little more effort, but the dividends are typically far more lucrative than just giving up because one table that was likely designed without much thought concerning archiving and operational data requirements, has reached a certain number of records.

It just seems implausible that a record count is the sole tipping point for a complete platform swap. There are exceptions, but my experience is that most teams get this wrong and quit long before the options have been exhaustively considered.

Huh. At the last DareTable conference I was thinking of topics for my next talk (whenever or wherever that happens), and one idea that I’d like to explore is the concept of where to put the business logic in an Airtable system. There are so many places—individual fields, formulas, rollup, view filters, scripts, apps, automations, forms, control/template records, and that doesn’t even touch on third party integrations.

And almost none of that business logic is accessible via API. Almost none of that business logic can be extracted from the UI and backed-up or documented easily. (Unless you consider a duplicate base to be an adequate backup.)

Hi Richard,

If a “small business” is generating such a lot of records, that could be a symptom that the Data Model isn’t (or ceased to be) quite right.

I strongly second Bill French’s advice. In particular, two points can be highlighted as probable areas of improvement:
· Data model: Logical restructuring in the over-populated tables.
· Archiving strategy: Backup methods. Logical compaction methods.

It’s interesting that you didn’t mention scripts. I’d assume that a base with that activity would have some scripts in operation — perhaps not enough or none at all? Sometimes, depending on the application architecture, a single script can have a sizeable impact on the amount of records, by making many of them unnecessary in the first place.

Please feel free to drop me a message, if interested in exploring a possible job opportunity to address this issue, while preserving your valuable investment in automations, formula, scripts et al.

Thank you for making my point.

You’re describing a climate where fracturing and atomizing the underlying solution’s logic is not only possible but encouraged. In fact, in many cases, Airtable’s shortfalls force you to do things that exacerbate the need to endlessly create workarounds and Goldbergian automation nightmares. Compounding the issue is the lack of API access to that which represents the core logic - ergo, it can never be documented without significant effort, so no one truly does it well.

This is not to say that Airtable hasn’t made some progress in the past few years, but it is without doubt, that any attempt to transition from Airtable to something else would be a memorably pleasant experience.

Thanks @Bill.French , you’ve prompted some thinking on our side about why we think we need those extra records - a lot of it is to do with keeping track of historical transactions for our customers - we have a recurring service model and are using a record for each iteration of the service. We want to make the entire service history visible to customers via a portal, and need to display various summary data across their entire history with us. … @kuovonne 's musings about collapsing data into text fields could be an option for us

There’s an undeniable and often irresistible urge to multi-purpose the same data from the same storage system. However, this is no different than placing the technical implementation details ahead of the business requirements. This is not uncommon, but it can lead to implementation biases that may not be desirable.

Historical data (like customer transactions) are generally static. Sure, there are times when they are adjusted for error corrections, but GAAP accounting principles frown on direct edits to historical accounting records. As such, data like this are generally considered “in lead”. If true, should such unchanging data be managed and served from a system designed for continual dynamic updates? It’s kind’a a waste of resources because it’s cold data that needs to be warmed from time-to-time. Airtable is keeping it “hot” all the time.

Indeed, there are also other options and these are implementation details, not business or technical requirements. I recommend paying particular attention to a deep assessment of the requirements without any consideration for the way you might implement those requirements. Then - and only then - decide how best to implement the requirements.

And to be clear, there are a number of business requirements that we often miss when examining archive, reporting, and customer transactions management. For example, analytics. What are the aggregation requirements for maintaining long-running analytics and KPIs for management which depend on future access to historical transactions? Collapsing data into text fields may solve one issue but create many more.