Soon You'll Need a CDN

Thank you Bill for documenting this subject a bit.

For “big projects”, I had never imagined delegating a CDN role to AIRTABLE since it would have been a design flaw from the start, it seemed logical to me, and more so since you had warned (as well as many other things) that one should really not design a project with such a starting bias.

My little prototypes did collect the URL of an Attachment Field (Image, PDF, JSON file) to consume it in another little prototype “built on top of airtable” but they are only little prototypes that I’m the only one to consume: I already knew from your previous warnings that it wouldn’t last.

Storing the References of Attachments housed in a real CDN (not a DRIVE) in my Tables instead of storing the Attachment itself is what I used to do before I discovered the ease of making it simpler in Airtable, and before I discontinued this practice inspired by your warnings.

As long as I can do it simply myself by means of my keyboard, my mouse, in my smaller bases.

To satisfy a programmatic, automation approach, I expect to have to struggle a bit with Script and CDN API but I have done it before and will do it again.

What bothers me a lot though, is the depreciation of the visual comfort of the Attachment Field in the AIRTABLE UI!

Imagine a FIELD populated with URLs of CDN instead of pretty images or PDF or .txt, .json files!

The loss is there.

I have imagined a possible solution:

Proxy: definition: the proxy of a Media File (Image or Video) is an Image compressed to lighten it while keeping it acceptably readable and manipulable.

This precision, although incomplete, seems to me sufficient to read and understand what follows.

Let’s imagine that for each image stored in a real CDN, I can get the URL of its Proxy because the CDN offers this possibility in its Pricing Plan.

This is a pure hypothesis but I have already seen it in very professional Media Storage & Management products rented by Companies.

Let’s imagine that by Airtable Script, I can query the CDN API to:

• List the Proxy Names stored in the CDN;


• Filter the Result by query or by parsing the response;


• Get the URL of the filtered Proxy based on its Name;


• Upload this Proxy in the Attachment Field of the Record whose Name (Primary Key Field) matches the name of the Proxy;
  
  (this is only a simplistic example without batch process, without exploitation of an index neither on CDN side, nor on Airtable side).
  
  Wouldn’t that be cool?
  
  Well, no, because it would only work randomly, since this feature of AIRTABLE has a frequent bias, very well documented by you in the Community, I thank you again.
  
  In the past and in the present, AIRTABLE has tried to find the bug and then ended up accepting it without fixing it.
  
  Let’s not make assumptions about the future but just a probability: it will not work in the near future.

So, what to do ?

AIRTABLE could offer us a new FIELD which would be a kind of I-Frame with restricted features for security reasons.

This Field would offer a URL property.

When we paste the URL of an Image Proxy stored in a real CDN, this Field would display a Preview of this Proxy in the table UI.

This is already possible visually using a Field Attachment but then the Proxy would be saved in Airtable, to put it more simply. (see details in your Message).

And this is only possible from my Keyboard and Mouse.

As the new Field I am proposing is a Display Component, to my mind there is no question of storing anything in Airtable’s internal CDN.

You paste the URL of your Proxy in the Field I propose, by hand or by Script,

and you get visually your proxy in your table UI.

Did I make myself clear ?

I’m thinking of drawing a mockup of all this in the current UI of Airtable if my proposal was not disqualified by the experts of the Community

because it would be issued from a biased thought or because a better inspired proposal than mine would replace it.

Fork me here please !

Thanks in advance.

olπ

It isn’t a new field type, but third party apps can do this to a certain extent. I believe Amplify can preview a wide variety of urls for a monthly subscription. My app Low Tech PDF can preview image URLs for a one-time flat fee.

What a wonderful observation. You are a visionary on par with the engineers at Coda.

To demonstrate how important this is, at my company we needed a utility app that would blend our image collection for a highway project with the ability to streamline the tagging of those images for more precise adjustments to our AI model. The images are in a secure CDN-of sorts but the database app itself must integrate them for rapid visual inspection and search.

We don’t use Airtable for this for precisely the reason you suggest - you can’t see or inspect the image in the database UI. So while storage by reference is an important design choice, Airtable eliminates the opportunity to enjoy the benefits of integrated image assets.

The app’s UI allows us to search, filter, and group AI data with a thumbnail view of the images and full view if needed. Buttons allow us to nudge the AI model with additional training biases. A custom pack updates our database in real-time which is always updating the AI model in real-time.

Why the purple tint? We use special cut filters on our image sensors combined with infrared light to see inside the vehicle even if it has really deep tinted glass.

How’d we do this? In Coda, a URL has the option of being rendered as a string or as the image it represents.

Yep, it’s a reality (in Coda).

This is precisely how they do it but most important, it doesn’t have to be URLs from their own CDN - it supports URLs from any source even computed Base64 values.

Perhaps, but it is certainly a new field behaviour; a requirement that seems obvious.

We’ve had conversations before about wanting to make fundamental changes to how fields work. One of these concepts was to have a clear separation between how a field’s value is stored and how it is displayed. This proposed change to how a url field type is displayed is just another extension of this idea.

Here’s a recap of my thoughts for anyone who might not have seen it before

• simplify the number of field types


• give each field type more extensive display options


• give each field a validation formula (preferably a formula that can reference other fields!)


• give each field conversion rules when non-conforming data is pasted or imported

For example

• a rating field could be a number field type with a validating formula that only accepts integers from 0-10, and display options that shows a user-specified character (emoji) the set number of times


• a currency field could be a number field with a validating formula that accepts numbers in a user defined range and display options that include a currency symbol and user defined symbols for the decimal and grouping symbol.


• a percent field is a number field with formation options that multiplied the number by 100 and tacks on a percent sign


• a phone number field is a text field with special formatting depending on the number of digits


• a url field could be a text field with a formula that optionally prefixes ‘https://‘ to the url if it is missing, and display options to render a thumbnail of the webpage (or the favicon, or a larger preview, or specific meta info, etc.)


• a text field could have formatting options to state if the display should be rendered as plain text, markdown, or html


• a date/time field could display options to always display in gmt, the local user time, or a set timezone determines by the field configuration (ugh, did I really say that? Timezones are confusing enough as is.)


• all of the formatting options should also be available to formula and rollup fields. Thus, a formula field could build a url that is then displayed as a preview of that url. A rollup field could be displayed as a fully formatted html page with line items.

But this would require a major overhaul of lots of existing code, a method for migrating existing data, gobs of testing, and a massive education campaign. As much as I would like to see this, I don’t think Airtable has the resources or inclination to do this right now. (Of course, I would love to be wrong about this guess.)

LOL! I love this one because y’all told me I was nuts when I suggested this in 2019. I complained endlessly in a very lengthy thread that a field is incapable of having both a value and a formula that optionally sets that value, thus making a “formula” type field necessary and all the baggage that comes with it.

100% agree.

Correct - the ship sailed and is unlikely to ever return.

I don’t recall saying you were nuts. I recall that I was unable to comprehend how this would work. You’ve taught me a thing or two since then. And to be clear, I’m talking two different formulas in my previous post: a data validation formula (to flag and/or reject invalid values), and a display options formula.

This makes me sad.

But since I’m dreaming, I’ll expand on the dream …

• simplify the number of field types


• give each field type more extensive display options, including a formula as part of the display options


• give each field a validation formula (preferably a formula that can reference other fields!) in order to reject or flag invalid data


• give default values for each field that works across all platforms (records created in the regular interface, via forms, and via automations / api / scripting / custom apps, etc.)


• give each field conversion rules when non-conforming data is pasted or imported


• give copy/export options for copying/exporting either the underlying data or the displayed data

Hi @kuovonne , Hi @Bill.French ,

I thank you for sharing these thoughts of yours and your conversations about the FIELDS because I had lost sight of them.

It was worth reading again, as far as I am concerned.

It is a pity that you are not Project Managers at AIRTABLE: AIRTABLE would be even better to say it briefly.

For some time, you have been dealing with :

• Data Validation ;


• Indexed Bases Search ;


• High End Reporting ;


• Automation ;


• Integration ;


• APIs ;


• Scripting ;

and suggesting workarounds, by illustrating your remarks with personal projects in AIRTABLE and/or in GOOGLE CLOUD, CODA.

It doesn’t solve the category of problems that should rather be prioritized internally at AIRTABLE but it inspires a lot, especially when you match them with some theoretical supplements like good practices versus troublesome practices to be avoided based on your professional experience, your contact with Customers, extended Projects but running every day in real life.

oLπ

Thank you for unveiling this part of your project.

You look inside the cockpit? :grinning_face_with_big_eyes:

You probably detect Humans asleep or busy on their Mobile Device instead of looking at the road.

In terms of Ai, it’s interesting: a Middle-Man facing your CODA’s UI tweaks the training of an Ai if I understood correctly.

I first had dreamed of this formula while experimenting with GitHub and Datasets offered to learn

because it seemed to me to be the learning solution taking advantage of the presence of a Human in front of the training experience of a model, before discovering it working in some AsAService business Ai-based product that popped-up on my phone’s instagram client.

CODA is used as a UI of a Database host, knowing you a little bit by reading you, in a Firebase Project:

it will have been necessary that I see CODA in this function at your place for me to register, loggedIn CODA and that I started to follow you there too.

Is your CODA querying your database with SQL expressions ?

In its case, how do you get the payload after querying it ?

Or REST queries (where you get JSON queries responses payload) ?

In August 2018, I simultaneously discovered Notion, Coda and Airtable in the same article

but it was something like Airtable I was looking for.

I bet on Airtable, fearing that the 3 would not survive, but only CODA remained in my toDo

partly because you use it by the way.

So I’m going to continue to explore CODA and its Community.

This will not exempt me from proposing the FIELD idea on Coda’s Way that I had just imagined on my side to AIRTABLE Staff but it’s for the act: they must have known for a long time that CODA has this and they must have their opinion on the subject: this FIELD still does not exist in their company despite all the interest that it presents, despite the discontinuation of the CDN role by Airtable.

oLπ

Sorry for derailing this thread. Here is a post that is a bit more related.

Airtable does not or want to be a CDN for accessing content outside of Airtable. CDN is an acronym for Content Delivery Network. A CDN is a way of making your file publicly available on the internet.

Another acronym that people may be interested in is DAM: Digital Asset Management. Like a CDN, a DAM makes your files available on the internet, but it also provides a lot of other services such as image transformation, optimization and tagging. If you need to move your files to a CDN, you might want to consider whether you should move them to a CDN service or a DAM service, if the extra features of a DAM would be useful.

Cloudinary is the DAM that I see referenced most often on these forums, but it is not the only one. Image Kit is another DAM with a free tier. There are several others, but I don’t remember their names off the top of my head. Anyone want to add to this list?

@kuovonne for sure, but they should fear hiring me for any role. :winking_face:

AI is a bit off-topic for this thread, but I’ll clarify - this project is not about safety; simply the measurement of HOV lane occupancy.

In AI, predictions are like a bell-curve; at the margins are mistakes and we use some specialized queries (other AI algorithms) to determine the probability the data is in error. This occurs when first training a model and when long-running systems are impacted by change. Humans are called in to suss out issues at the ends of the curve. All other learning is automated.

In this example, correct. The UI is issuing requests to a very large data set in ElasticSearch via a custom Pack.

No. Another machine-learning system tells us specifically which data set is needed from ElasticSearch.

I think you made a pretty good bet. Have you noticed how much better and quickly Airtable had progressed lately?

Vendors do not have the liberty of copying ideas without some level of patent investigation. It’s a scary time when a firm decides to copy one vendor’s approach only to learn that another vendor (with billions) may have a claim to that approach. As such, we cannot always level blame at Airtable in every case.

To be clear, Airtable’s CDN is not being discontinued; its “role” as you point out has been slightly narrowed and for good reason. The future design is both fluid, more secure, and will provide massive benefit. It simply won’t support the broader whims of developers who are looking for a free CDN for $24/month. :winking_face:

In my view, the CDN policy change is irrelevant to the idea that fields should provide more abstract features such as the ability to render content from any legitimate source - e.g., @kuovonne’s observations. Long before Airtable decided to limit its CDN role, they should have been pondering ideas like embedding external images or videos (for example) in its new Interfaces feature as well as data records. It makes sense and it’s seemingly an obvious business requirement. Databases, after all, are storehouses of meta information and that meta information should not be constrained to artefacts that are physically native to the data.

Bingo! Well said. And if this is the case, Airtable should also consider that its platform should accommodate pathways for content that is OUTSIDE Airtable based on references INSIDE Airtable.

DAM: Digital Asset Management.

Thanks for these details Kuovonne: it will certainly clarify the offer to people who haven’t seen this as I have seen it in one or the other big audiovisual production center that buy / rent big DAMs seen at NAB or IBC: the big annual shows of this profession.

These additional services also justify the 12 month price tag that is high at Cloudinary.

I knew Cloudinary for the same reason as you but I just went to Image Kit whose free third party seems more interesting for modest prototypes.

Yes, that would be cool.

I will definitely do the same after a personal experience.

oLπ

I’ve read it all as I’m looking for an effective way to have the best of both world:

• Attachment field within Airtable to easily visualize/edit attachments


• Attachment CDN url field to easily resolve the url of an asset when querying using the Airtable REST API (stored into a dedicated CDN, not Airtable’s)

The main post is really enlightening about the change, but doesn’t provide an actual solution, that’s what I’m looking for.

And, as @Bill.French pointed out, it’s definitely complicated.

Not only do we need to detect when an attachment is added/modified for a record, we also need to send that attachment to our own CDN, retrieve its link, and store it in an “attachment CDN url field” (basically a URL field).

So, the whole process can be seen as:

1. Detect attachment change


2. Send attachment to CDN


3. Store CDN asset’s url into Airtable

I’m thinking about the first step, detecting the change. This only seem very difficult using Airtable automations, I don’t even see a way to detect changes for all the base’s attachments in a simple way, it’d require either a different automation for each field, or maybe rely on an App/Extension that watches changes from all tables to trigger the rest of the workflow. But, AFAIK, Apps must be executed manually, and it wouldn’t satisfy my need for a fully automated system.

So, I’m kinda stuck at step 1. Any better idea?

Maybe tools like Make, n8n, can help differently, to make this process scalable to configure/maintain.

Ideally, I’d easily configure a mapping of all attachment fields to watch for (those for which I need to store into a dedicated CDN), and the name of the field where to store the CDN url.

Represented as JS, it would look like:

{

  field: 'image',

  destination: 'imageCDNUrl', // Contains the CDN link of the "image"

}

Even more ideal would be to have this built-in within Airtable.

The ability to specify, for any “Attachment” field to enable the public CDN for that field, and in which field to store the CDN url would be, by far, the simplest way to deal with the upcoming nightmare.

Files stored this way would be properly stored into a public CDN, and would have its own pricing and limits based on the plan. Airtable could imagine something like “pay-as-you-go” where customers would pay additionally based on the number of CDN assets stored, this kind of stuff.

It would make sense to propose such “public CDN” feature, which would be a good complement to the “internal CDN” that can only be accessed from within Airtable. Assuming it has its own pricing and managed differently, it would both provide great value, while forcing customers to only enable the public CDN for the few fields we need to have public. It wouldn’t be a hindrance to security, GDPR, or performances.

I really hope such a system will be designed, better sooner than later, it would help with the upcoming transition quite a lot. :crossed_fingers:

{

  field: 'image',

  destination: 'imageCDNUrl', // Contains the CDN link of the "image"

}

Correct. But, if you are trying to create this process in enterprise accounts, the Enterprise Events API will achieve this.

I believe Airtable may have changed its position concerning the REST API’s ability to retrieve the latest signed URL. @kuovonne recently pointed out that this passage was updated to include a reference to the REST API.

Any attachment URLs obtained via the public API and the CSV export functionality will expire after a few hours.

That’s good to know, thanks for pointing this out.

Unfortunately, it won’t help me. As we use SSG technology (Static Site Generation), where the site is generated once as static files (JS/HTML/CSS) and then left as-it. If the links are valid at the time of the generation, they’ll eventually become invalid, breaking all images on our beautiful websites.

And we aren’t using the Enterprise plan either, so the Events API will not be of help in our case.

How did you automate the whole process? You mentioned having done it a couple of times.

Could you please describe the tools stack you used, by any chance?

In both cases, I found that the pace of change on records with attachment fields was sufficient to fire off an automation that always attempts to keep a Firestore database apprised of existing or new attachment artefacts. Using a Firebase Cloud Function, I was able to sustain a replica of all Airtable attachments, including attachments that were modified or added thereafter.

The prime task of the automation script is to inform Firebase that something has changed and it may concern an attachment field. The cloud function serves as a webhook listener that needs only the Airtable record ID and makes an immediate call-back into the Airtable REST API to harvest all attachment data for that record.

The webhook listener then dispatches other methods to ensure that the latest attachment documents are instantiated in the CDN (a Firebase storage container which is really two components - a real-time data store of CDN links and the actual documents/images/etc).

Okay, I see. That’s definitely not beginner level. And not scalable either (you need one automation for each attachment field you want to have a public url).

Thanks, it helps.

Not true. You can create automations that sweep up all records with change events based on many fields. Remember - in this architecture, you need only inform the webhook listener that something changed in a given record. Then it’s up to the webhook listener to suss out the changes that cause its CDN to capture and host the content.

If by “scale” you mean a hundred thousand attachments all maintained by a single automation, this is possible because I’ve done it.

You’re right, I was internally thinking of different Attachment fields from different tables, those will need one dedicated automation, for each table.

Also, AFAIK the “create” and “update” events are different, so you might need to have one automation which triggers during the “create record” event, and another which triggers during “record update” while watching all the Attachment fields.

So, in this case, that would require 2 automations per table where you’d want to make any attachment public. Associated with the 50 automations limit, it can be a hindrance (sure, you could have them in another database, but there are other issues to think of when doing that).

And it makes things harder to maintain as well.

I’ve opened the Feature request: Public CDN opt-in for attachments feature request, I believe it’d be the simplest way to deal with all of that.

Once again, this can be overcome and simplified. Imagine a view that sweeps up all records based on the value of a field that tracks the last modified date/time. This allows you to see both created and modified records which trigger the webhook listener.

FYI, very little of what Bill posts is beginner level.

Well, there are some exceptions; I post beginner-level questions when I’m stuck. :winking_face:

Hi @Bill.French ,

Your solution overall-diagram






is inspiring me for a while but as you know, airtable + js + integrations is not my first job,

so I’m delaying the moment I would jump into your favourite Firebase / Firestore you were still talking about e.g. around airborne, elastic search, indexed full-text search and inside other threads.

Between eggs+roasted bacon and coffee (this is my actual usual breakfast, even in Belgium),

I was bookmarking something that could help to browse and maintain some pictures and light-weight (in my useCase) videos , not CDN’ed by airtable, but by something called a “firebase project” that’s still dark stuff for me (but not yet a dark hole :winking_face: )

I can’t wait to go from a not yet very well founded wish to a well conducted learning experience but I still don’t get time available to cross this new milestone.

So it is still possible that it will never happen before the deadline of 1/11/2022…

You (and the Community) might think that I would take the risk of desynchronizing the Airtable Attachments, the CDNurl’s and the Firebase project by going to manipulate it behind the back of Airtable and the automation yet discussed today by using ROWY or something like that if it exists, but it’s not so!

I’m just so newbie in Firebase that it would be very comfortable for me to see my records and possibly help me when my workflow prototype wouldn’t work as expected, which will happen.

Disclaimer: although I’m bringing ROWY up in this thread, I emphasize that this is strictly within the scope of this thread and is not an invitation to drift away by trying to compare ROWY and Airtable, at least not here.

If you are interested in this discussion, please open another thread elsewhere in the forum, thank you very much!

Yum.

The truest test of a vendor’s confidence is seeing how they react when you openly link to names of perceived competitors in their own forum. :winking_face: