Scalable migration?

Richard_Foxwort · ‎Apr 30, 2022

Hi all, Sadly, I need to migrate away from Airtable due to the lack of a scale plan that makes any sense.

I’m a Pro user and we are right up against the 50,000 record limit. It has been extensively discussed elsewhere on this forum that unfortunately Airtable does not offer a sane upgrade licensing next step, and for me there’s nowhere to go but out to market to find a new platform. The support team actively advised against the Enterprise plan because we only need more records (lots more records!) not more users or more bases. The enterprise plan cost is laughable anyway for a small business like ours, and would only provide us with additional 50k records, which we’d outgrow in several months anyway.

So I’m looking for a much more scalable alternative that can offer some or most of the user friendliness of Airtable, and can support a simple(ish) migration. My base has quite a lot of automations, plus formula and lookup fields, so I know this is a big ask…

I found Seatable which claims to support most (all?) Airtable features, plus quite a bit more, and appears to also offer a fairly simple migration path from Airtable.

Has anyone tried this for real with a fairly complex ~50,000 record base? I’m keen to understand what issues/gotchas others may have encountered to help make a decision if this really is a viable way forward.

Thanks in advance

ScottWorld · ‎May 01, 2022

Just a few days ago, Airtable increased the number of records in their enterprise plan to 250,000 records per base, which is a great step in the right direction from them. (Their pricing webpage hasn’t been updated to reflect this official change yet.)

With 250,000 records per base, perhaps you could create some sort of a record archiving system in Airtable to stay within each base’s limits?

But if 250,000 records per base still won’t be enough, I always highly recommend that everyone who outgrows Airtable to migrate to Claris FileMaker Pro (soon to be rebranded as Claris Pro), which is enterprise-level database software that has almost no limits at all. You can have unlimited records, unlimited automations, unlimited integrations, etc.

I was a FileMaker developer for 30 years. I’ve retired from FileMaker development to become a professional Airtable consultant, but if you reach out to me through my website, I can turn you onto some excellent FileMaker developers who can help you with your migration:

Bill_French · ‎May 01, 2022

I have been asked to render opinions concerning the situation you find yourself in today and in almost every case (11 of them so far), the 50,000 record limit was determined without deep consideration of the context. In 9 of 11 cases, they had only one table that was near 50,000 rows and they assumed there was nothing they could do except leave Airtable. In 7 of those 9 cases, they remained on Airtable because they eventually realized that:

Their data model wasn’t well-optimized.
They had failed to identify the true nature of operational vs strategic data.
They could not envision an archive strategy that would continue to sustain their reporting objectives.

Are you possibly like the seven I have encountered?

Being up against the record limit is contextual. In my experience, the context that triggers a wholesale replacement of a perfectly competent platform is often misunderstood or not exhaustively researched before jumping ship.

Formula fields in Airtable alone are a unique beast that doesn’t translate well to any other platform.

I also have some experience with SeaTable and it is quite powerful in some ways and lacking in many because it doesn’t yet have the customer base needed to refine the platform. Changing from Airtable to just about everything will be unpleasant.

kuovonne · ‎May 01, 2022

I’m guessing that you used at least three different techniques across those seven clients. I can guess at two possible techniques

collapse selective data into json in long text fields, and only expand it when needed
store data outside of Airtable and fetch it only when needed

I think you’ve written up about those techniques before, but I don’t recall the exact posts. I’d also be curious about any other tricks you use.

That’s quite the complement to Airtable.

Bill_French · ‎May 01, 2022

Ha ha - not intended to be a compliment. Precisely the opposite actually. Airtable lulls complacency when it comes to database and app design. The formula field is a good example; most database systems do not have such fields of calculation type. SeaTable is at least one exception. But we tend to do a lot of things in Airtable to make stuff work that we would not do in other systems.

By “unpleasant” I’m really referring to the massive business logic that ends up in tables and fields, whereas, other competitors often have a different vision for implementing business logic. And they only need to be just a little different to make the transition very unpleasant.

They’re not tricks - just good modelling can often optimize storage and with a well-designed archiving approach, you can remain 100% in Airtable. Yes, it’s a little more effort, but the dividends are typically far more lucrative than just giving up because one table that was likely designed without much thought concerning archiving and operational data requirements, has reached a certain number of records.

It just seems implausible that a record count is the sole tipping point for a complete platform swap. There are exceptions, but my experience is that most teams get this wrong and quit long before the options have been exhaustively considered.

kuovonne · ‎May 01, 2022

Huh. At the last DareTable conference I was thinking of topics for my next talk (whenever or wherever that happens), and one idea that I’d like to explore is the concept of where to put the business logic in an Airtable system. There are so many places—individual fields, formulas, rollup, view filters, scripts, apps, automations, forms, control/template records, and that doesn’t even touch on third party integrations.

And almost none of that business logic is accessible via API. Almost none of that business logic can be extracted from the UI and backed-up or documented easily. (Unless you consider a duplicate base to be an adequate backup.)

Grunty · ‎May 01, 2022

Hi Richard,

If a “small business” is generating such a lot of records, that could be a symptom that the Data Model isn’t (or ceased to be) quite right.

I strongly second Bill French’s advice. In particular, two points can be highlighted as probable areas of improvement:
· Data model: Logical restructuring in the over-populated tables.
· Archiving strategy: Backup methods. Logical compaction methods.

It’s interesting that you didn’t mention scripts. I’d assume that a base with that activity would have some scripts in operation — perhaps not enough or none at all? Sometimes, depending on the application architecture, a single script can have a sizeable impact on the amount of records, by making many of them unnecessary in the first place.

Please feel free to drop me a message, if interested in exploring a possible job opportunity to address this issue, while preserving your valuable investment in automations, formula, scripts et al.

Bill_French · ‎May 02, 2022

Thank you for making my point.

You’re describing a climate where fracturing and atomizing the underlying solution’s logic is not only possible but encouraged. In fact, in many cases, Airtable’s shortfalls force you to do things that exacerbate the need to endlessly create workarounds and Goldbergian automation nightmares. Compounding the issue is the lack of API access to that which represents the core logic - ergo, it can never be documented without significant effort, so no one truly does it well.

This is not to say that Airtable hasn’t made some progress in the past few years, but it is without doubt, that any attempt to transition from Airtable to something else would be a memorably pleasant experience.

Richard_Foxwort · ‎May 03, 2022

Thanks @Bill.French , you’ve prompted some thinking on our side about why we think we need those extra records - a lot of it is to do with keeping track of historical transactions for our customers - we have a recurring service model and are using a record for each iteration of the service. We want to make the entire service history visible to customers via a portal, and need to display various summary data across their entire history with us. … @kuovonne 's musings about collapsing data into text fields could be an option for us

Bill_French · ‎May 03, 2022

There’s an undeniable and often irresistible urge to multi-purpose the same data from the same storage system. However, this is no different than placing the technical implementation details ahead of the business requirements. This is not uncommon, but it can lead to implementation biases that may not be desirable.

Historical data (like customer transactions) are generally static. Sure, there are times when they are adjusted for error corrections, but GAAP accounting principles frown on direct edits to historical accounting records. As such, data like this are generally considered “in lead”. If true, should such unchanging data be managed and served from a system designed for continual dynamic updates? It’s kind’a a waste of resources because it’s cold data that needs to be warmed from time-to-time. Airtable is keeping it “hot” all the time.

Indeed, there are also other options and these are implementation details, not business or technical requirements. I recommend paying particular attention to a deep assessment of the requirements without any consideration for the way you might implement those requirements. Then - and only then - decide how best to implement the requirements.

And to be clear, there are a number of business requirements that we often miss when examining archive, reporting, and customer transactions management. For example, analytics. What are the aggregation requirements for maintaining long-running analytics and KPIs for management which depend on future access to historical transactions? Collapsing data into text fields may solve one issue but create many more.