That’s not inefficient — that’s how a database works! You can always create a view which hides the notes field if you don’t want to see it, but that would probably make things even more confusing.

Really? I’m no expert but I don’t think that is how all databases work. It seems pointless to have a column that will have tons of empty fields. Why not just store the data that’s wanted?

I feel like there is a 2 table solution here, I just don’t know what it is. If someone knows pls let me know!

So you want to create ANOTHER table with ANOTHER field, and then potentially link it to your first table for yet ANOTHER field (a lookup field)? Sounds super efficient.

If this is your motivational speaking technique in action then I would suggest you go back to the drawing board! :cowboy_hat_face:

Ha! I wasn’t trying to motivate you there, I was trying to stop you from overthinking the solution which you already figured out.

Now THIS is some of my motivational speaking technique in action:

Perhaps you can create a magic question about your database!

I’ll watch your vid later…with a gin & tonic to take off the edge.

Returning to the problem…I contend I was not overthinking. I was merely offering what really amounts to a hack by way of illustrating I thought about the problem. I am sure there is someone who knows Airtable who could offer the “correct” way to do what I describe, rather than adding a mostly empty column to a table.

Perhaps — let’s see what ideas other people come up with!

Well, there you have it :smiling_face_with_halo:

So the best solution is to add the extra column that contains mostly empty fields? Just trying to get to the right way of doing things!

Go ahead and add the extra column that will be empty for most records.

One of the beauties of Airtable is that it is easy to change the configuration of the base if you later decide on a different system.

While having “rivers” of empty cells is often an indication that you should examine the base schema, it does not always indicate a problem.

Thank you for stating a clear non-sarcastic solution.


Are you a motivational speaker by any chance???

Except that my very first response was very clear and non-sarcastic. Try looking through the thread to see where the snark started, and it didn’t start with me. :man_shrugging:

It started here. Look inward, Mr Motivation.

Which was in reply to your discourteous message here:

If you’d like high quality advice from the Airtable experts around here who are volunteering their time to help you, it would probably be helpful to be warm & gracious.

If you think that was discourteous you need to get out more often. It was sincere and honest probing, looking for the best solution.

You seem very easily triggered. Lapsing into a defensive and sarcastic safespace might make sense to you but it will not work with the overwhelming majority of people, a lot of whom I imagine could use some motivation to get their lives on track. This advice comes with no charge to you.

Let’s keep it friendly here guys :face_without_mouth:

@Heirtable, as @ScottWorld and @Heirtable are telling you the same thing, both being unbelievable Airtable experts, do you feel you have the answer to your question?

I’ll take a stab at this. :winking_face: But from a different perspective.

Imagine a database architecture where there are never “empty” fields. In fact, an underlying storage architecture that physically does not have any fields in a record unless there is data in those fields. This is the nature of a NoSQL database; it uses no storage or resources to store and manage null values, and null values are treated very different than empty strings and numbers. While the client may render such non-existent fields as if they exist, they really do not.

Optimization Proof

If you query a given record using the Airtable API, you will quickly learn that fields that have no values are simply not in the result set.

Do you still think it’s inefficient?

To be clear, this happens when you query Airtable’s REST API. When you query for a record using the Scripting or Custom Apps API, null field values are included as nulls.

The absence of the field in the query result from the REST API is not proof one way or another of how the underlying data is stored.

——

I suggest yet another point of view. Instead of thinking of what is the most efficient method of storing the data, what method of arranging the data best fits your workflows?

If having a column in the same table makes data entry quick and convenient, despite the large number of empty cells, do it.

If seeing lots of empty cells makes you uncomfortable, hide the field so you only see it in expanded view. Or put the data in a linked table if you don’t mind the extra work in dealing with a linked table. (However, if having lots of empty cells in Airtable bothers you, does having blank columns and rows in a spreadsheet also bother you?)

Love it. This is true, but it intimates the likely underpinning, correct? If this were not true, do you think the API team would go to the trouble to make it look like the underlying architecture was efficient when it really wasn’t? :winking_face:

I think we both know that Airtable puts effort into storing data efficiently, even if we don’t know the full, exact details.

On the other hand, there could be a different, simple explanation of why the REST API omits fields with no data. It could be omitting the data simply to reduce internet bandwidth.

Similarly, if you query the scripting API, null fields are returned in the result. This is not Airtable trying to expose or obfuscate how the underlying data is stored. Rather, this is Airtable trying to make things easier for the script writer, when you do not have the same internet bandwidth issues.

Indeed, and this is likely because both of these SDKs are implemented at the client, not the server, right? Ergo, I suspect they had to manicure the results for their own client. API developers must do the same - we have to accommodate the fact that the values are undefined and interpret them as null (not empty) values. But, the lack of such values at the lowest level interface pretty much indicates there is no wasted space.

I agree that the difference in the APIs is due to the client/server situation. And the REST API is certainly an older system. And I totally agree that there is no wasted space.

However I don’t believe that the REST API is necessarily the best representation of how Airtable currently stores its underlying data. The field values for some of the more complex data types is very different between the APIs, and I suspect that the scripting API value is closer to what is actually stored.

But we are really getting off into a tangent here. I wonder if Heirtable is upset by us hijacking the thread. I hope not. Side chats like this, and getting to read everyone’s particular writing style is part of what makes this community so vibrant and fun.

I feel exactly the opposite. Here’s why…

#1 There is only one publicly-available API (the REST API). The others that you refer to as “APIs” are actually SDKs, and SDKs are built on top of APIs - which API exactly is unknown because Airtable likely has an internal one that we aren’t privy too. However, whatever API it is, it is the most intimate one can get with the platform. SDKs are lby definition] abstraction layers.

#2 The SDKs exist in Airtable at the client; not the server. And since they are client-bound, there’s a strong likelihood that they were designed to be helpful in building clients. After all, this is precisely their intention and any such “helpers” baked into this abstraction layer are probably going to represent not the underlying data architecture, but a transformed data layer suitable for efficient rendering and other UI logic.

That said - whatever guesswork we may engage in is far more likely to be accurate if we let the result sets of the API guide us because the SDKs must be more abstract to achieve what they do. Ergo, we can safely surmise that they are augmented realities of the underlying database architecture.

Way Off Point?

To be clear, I don’t write for just the questioner; I try to be broadly informative to the 10,000 readers who will follow @Heirtable’s inquisitiveness. But, I think we’re still on point - the question was clear - aren’t empty fields a waste of resources? Pretty sure we can agree, they’re not.

However… (love that word)

To really address this fully, we need to circle back to one of your earlier points concerning the debate over flat tables vs relational models. I think there’s at least a small book we need to hammer out. You first. :winking_face:

I see the word “API Reference” as a heading for the Scripting documentation and Custom Apps documentation so often that I tend to think of them as APIs. But SDK is a better term.

I don’t see why the publicly REST API is necessarily closer to what is actually stored versus the SDKs. The public REST API is not the API that Airtable itself uses. (I remember getting a public statement from Airtable to this effect when custom blocks came out.) There is a strong likelihood that the REST API was designed to be helpful for consumers of the API. For example, in the REST API single select fields are returned as strings, but in the SDKs they are returned as objects with an id, color, and name. The id of the choice is probably stored (or why bother having an id?), but that internal id is pretty useless to the consumer of the REST API, so only the name is returned.

I don’t write for just the questioner either. But I think you vastly overestimate the number of readers of an individual thread. But given the original poster’s response to earlier posts in this thread, I still wonder.

Oh no. I’m afraid to answer this. I currently firmly believe that when building an Airtable base, the base designer should take into account the inherent structure of the data, the abilities & limitations of the platform, the workflows & preferences of the users. Sometimes this will mean a flat table. Other times it will mean linked tables.

Sometimes people use flat tables when a linked table would be better because they need to deal with record count limits. Other times people use linked records when the data could be flat, because having multiple tables/tabs makes more sense to them and fits better with their workflows or security models.

I was about to compose an op until after reading yours I realised I have. the same or similar issue.

I am a commercial property surveyor and currently I use Filemaker 12 as my main database for property information but I want to transfer all the records, some 80,000, to AT. On an AT base I have fields that generally replicate the FM fields but I shall not. be importing records but copy and paste one by one because for me the advantage of a new system is to have a. rethink. For example, in FM I have 5 fields for a particular aspect whereas in AT only one because I’ve customised the field to Multiple Select.

In my FM database I have created what appears to be a table (with label ‘Rent History’) consisting of 12 fields and 10 rows. Reading this thread I now realise that it is not a table as such but 120 fields that I have positioned on the FM layout to display the content in each field as if a table. I would guess that relatively few records have enough content to occupy most of the fields, the bulk are empty.

Where AT differ it seems to me is in layout. Unless I use the Page Designer app for a layout or have a view for each property (which might not be feasible, but would be inefficient) I am not going to be able to know until entering the content which fields should be hidden. Moreover, as content is not always fixed, it would be better not to hide any as having to unhide would be a waste of time as and when.

After several days pondering my conclusion is to have 12 fields, each customised type to Long Text. Instead of ‘rows’ I shall enter the content in each field as if a row. For example:

Rent History 1


RR 2016-09-29 - £8000 Zone A £20 Memo ? ML

Rent History 2


RR 2019-09-29 - £10,000 Zone A £25. Memo 2019-10-06 ML

Rent History 3


RR 2021-09-29 - £10,000 Zone A £25. Memo 2021-10-29 ML

The downside of my conclusion is how to manipulate the data. That is resolved by having other fields for data that I want to manipulate. Not ideal but a work-around.

What do others think?