Re: Soon You'll Need a CDN

Bill_French · ‎Apr 28, 2022

One of the best features of Airtable is the ability to drag images, files, and even videos (i.e., digital artefacts) right into an attachment field. When you do this, Airtable makes a copy of your image and stores metadata about the image into your base. It also quietly moves the copy of your image to its content delivery network (CDN). The URL to your image has been openly accessible providing a wonderful way to share access to your assets For almost eight years, this has been possible but that’s going to change in significant ways later this fall.

Long Time Coming

I predicted this would happen in 2019. They warned us then and they are giving you almost a year to make plans. A lot of folks complained here, but this change is most certainly going to happen and it is in everyone’s best interest.

Now that Airtable has officially announced new specifications for its CDN, any solutions that depend heavily on images that need to be accessed outside the Airtable UI will probably require that you host them in your own CDN.

What’s a CDN?

Cloudflare provides one of the better definitions:

A content delivery network (CDN) refers to a geographically distributed group of servers which work together to provide fast delivery of Internet content. A CDN allows for the quick transfer of assets needed for loading Internet content including HTML pages, javascript files, stylesheets, images, and videos. The popularity of CDN services continues to grow, and today the majority of web traffic is served through CDNs, including traffic from major sites like Facebook, Netflix, and Amazon.

Airtable, of course, has its own CDN to provide access to the images and files you upload to attachment fields. The intent of this CDN is for it’s paying customers; not as a global hosting server. If you need to host your Airtable-bound digital artifacts, you’re going to need to essentially replicate Airtable’s CDN, so you might want to read on.

Thanks Google Drive!

One might think that Google Drive, or other services such as DropBox or Box will host these files for me, but that could be a mistake. These services don’t want to be your CDN any more than Airtable does. They include ToS clauses that allow them to throttle open access to your documents despite being fully shared to the open Internet.

One Approach

There are many ways to create a CDN for your attachment files. In fact, Cloudinary is an ideal ready-made CDN that also provides many features for sizing and rendering images in custom formats and styles. However, it’s not a magic bullet and the pricing might be a little steep depending on your needs.

The complex part requires that you create some sort of an integration that pulls (or pushed) your attachments from Airtable (perhaps using the API or script automation) and upload yet another copy into your CDN environment. If you have a serious commitment to attachments, this could be a sizeable challenge. There’s also the issue of keeping your CDN updated in near-real-time, a key requirement that is more easily addressed with Firebase.

These complexities may not be easy to build depending on your selection of a premade SaaS CDN solution. And, you might also get lucky with Zapier or another glue-factory approach. Experiment a little and see what works best for your situation.

I have already helped a few friends design an approach that uses Firebase, a Google Cloud Platform service that is very fast and provides a generous free tier and an integrated CDN.

WARNING - CDNs are not universal in features. Some are globally distributed while others provide localized servers in your country. The costs vary greatly as well. And it also depends on your hosting and delivery requirements.

kuovonne · ‎May 01, 2022

Sorry for derailing this thread. Here is a post that is a bit more related.

Airtable does not or want to be a CDN for accessing content outside of Airtable. CDN is an acronym for Content Delivery Network. A CDN is a way of making your file publicly available on the internet.

Another acronym that people may be interested in is DAM: Digital Asset Management. Like a CDN, a DAM makes your files available on the internet, but it also provides a lot of other services such as image transformation, optimization and tagging. If you need to move your files to a CDN, you might want to consider whether you should move them to a CDN service or a DAM service, if the extra features of a DAM would be useful.

Cloudinary is the DAM that I see referenced most often on these forums, but it is not the only one. Image Kit is another DAM with a free tier. There are several others, but I don’t remember their names off the top of my head. Anyone want to add to this list?

Bill_French · ‎May 01, 2022

@kuovonne for sure, but they should fear hiring me for any role. :winking_face:

AI is a bit off-topic for this thread, but I’ll clarify - this project is not about safety; simply the measurement of HOV lane occupancy.

In AI, predictions are like a bell-curve; at the margins are mistakes and we use some specialized queries (other AI algorithms) to determine the probability the data is in error. This occurs when first training a model and when long-running systems are impacted by change. Humans are called in to suss out issues at the ends of the curve. All other learning is automated.

In this example, correct. The UI is issuing requests to a very large data set in ElasticSearch via a custom Pack.

No. Another machine-learning system tells us specifically which data set is needed from ElasticSearch.

I think you made a pretty good bet. Have you noticed how much better and quickly Airtable had progressed lately?

Vendors do not have the liberty of copying ideas without some level of patent investigation. It’s a scary time when a firm decides to copy one vendor’s approach only to learn that another vendor (with billions) may have a claim to that approach. As such, we cannot always level blame at Airtable in every case.

To be clear, Airtable’s CDN is not being discontinued; its “role” as you point out has been slightly narrowed and for good reason. The future design is both fluid, more secure, and will provide massive benefit. It simply won’t support the broader whims of developers who are looking for a free CDN for $24/month. :winking_face:

In my view, the CDN policy change is irrelevant to the idea that fields should provide more abstract features such as the ability to render content from any legitimate source - e.g., @kuovonne’s observations. Long before Airtable decided to limit its CDN role, they should have been pondering ideas like embedding external images or videos (for example) in its new Interfaces feature as well as data records. It makes sense and it’s seemingly an obvious business requirement. Databases, after all, are storehouses of meta information and that meta information should not be constrained to artefacts that are physically native to the data.

Bill_French · ‎May 01, 2022

Bingo! Well said. And if this is the case, Airtable should also consider that its platform should accommodate pathways for content that is OUTSIDE Airtable based on references INSIDE Airtable.

Olpy_Acaflo · ‎May 02, 2022

DAM: Digital Asset Management.

Thanks for these details Kuovonne: it will certainly clarify the offer to people who haven’t seen this as I have seen it in one or the other big audiovisual production center that buy / rent big DAMs seen at NAB or IBC: the big annual shows of this profession.
These additional services also justify the 12 month price tag that is high at Cloudinary.

I knew Cloudinary for the same reason as you but I just went to Image Kit whose free third party seems more interesting for modest prototypes.

Yes, that would be cool.
I will definitely do the same after a personal experience.

oLπ

Ambroise_Dhenai · ‎Aug 18, 2022

I’ve read it all as I’m looking for an effective way to have the best of both world:

Attachment field within Airtable to easily visualize/edit attachments
Attachment CDN url field to easily resolve the url of an asset when querying using the Airtable REST API (stored into a dedicated CDN, not Airtable’s)

The main post is really enlightening about the change, but doesn’t provide an actual solution, that’s what I’m looking for.

And, as @Bill.French pointed out, it’s definitely complicated.

Not only do we need to detect when an attachment is added/modified for a record, we also need to send that attachment to our own CDN, retrieve its link, and store it in an “attachment CDN url field” (basically a URL field).

So, the whole process can be seen as:

Detect attachment change
Send attachment to CDN
Store CDN asset’s url into Airtable

I’m thinking about the first step, detecting the change. This only seem very difficult using Airtable automations, I don’t even see a way to detect changes for all the base’s attachments in a simple way, it’d require either a different automation for each field, or maybe rely on an App/Extension that watches changes from all tables to trigger the rest of the workflow. But, AFAIK, Apps must be executed manually, and it wouldn’t satisfy my need for a fully automated system.

So, I’m kinda stuck at step 1. Any better idea?
Maybe tools like Make, n8n, can help differently, to make this process scalable to configure/maintain.

Ideally, I’d easily configure a mapping of all attachment fields to watch for (those for which I need to store into a dedicated CDN), and the name of the field where to store the CDN url.

Represented as JS, it would look like:

{
  field: 'image',
  destination: 'imageCDNUrl', // Contains the CDN link of the "image"
}

Even more ideal would be to have this built-in within Airtable.

:point_right: The ability to specify, for any “Attachment” field to enable the public CDN for that field, and in which field to store the CDN url would be, by far, the simplest way to deal with the upcoming nightmare.
Files stored this way would be properly stored into a public CDN, and would have its own pricing and limits based on the plan. Airtable could imagine something like “pay-as-you-go” where customers would pay additionally based on the number of CDN assets stored, this kind of stuff.

It would make sense to propose such “public CDN” feature, which would be a good complement to the “internal CDN” that can only be accessed from within Airtable. Assuming it has its own pricing and managed differently, it would both provide great value, while forcing customers to only enable the public CDN for the few fields we need to have public. It wouldn’t be a hindrance to security, GDPR, or performances.

I really hope such a system will be designed, better sooner than later, it would help with the upcoming transition quite a lot. :crossed_fingers:

Bill_French · ‎Aug 18, 2022

Correct. But, if you are trying to create this process in enterprise accounts, the Enterprise Events API will achieve this.

I believe Airtable may have changed its position concerning the REST API’s ability to retrieve the latest signed URL. @kuovonne recently pointed out that this passage was updated to include a reference to the REST API.

Any attachment URLs obtained via the public API and the CSV export functionality will expire after a few hours.

Ambroise_Dhenai · ‎Aug 18, 2022

That’s good to know, thanks for pointing this out.

Unfortunately, it won’t help me. As we use SSG technology (Static Site Generation), where the site is generated once as static files (JS/HTML/CSS) and then left as-it. If the links are valid at the time of the generation, they’ll eventually become invalid, breaking all images on our beautiful websites.

And we aren’t using the Enterprise plan either, so the Events API will not be of help in our case.

How did you automate the whole process? You mentioned having done it a couple of times.
Could you please describe the tools stack you used, by any chance?

Bill_French · ‎Aug 18, 2022

In both cases, I found that the pace of change on records with attachment fields was sufficient to fire off an automation that always attempts to keep a Firestore database apprised of existing or new attachment artefacts. Using a Firebase Cloud Function, I was able to sustain a replica of all Airtable attachments, including attachments that were modified or added thereafter.

The prime task of the automation script is to inform Firebase that something has changed and it may concern an attachment field. The cloud function serves as a webhook listener that needs only the Airtable record ID and makes an immediate call-back into the Airtable REST API to harvest all attachment data for that record.

The webhook listener then dispatches other methods to ensure that the latest attachment documents are instantiated in the CDN (a Firebase storage container which is really two components - a real-time data store of CDN links and the actual documents/images/etc).

Ambroise_Dhenai · ‎Aug 18, 2022

Okay, I see. That’s definitely not beginner level. And not scalable either (you need one automation for each attachment field you want to have a public url).

Thanks, it helps.

Bill_French · ‎Aug 18, 2022

Not true. You can create automations that sweep up all records with change events based on many fields. Remember - in this architecture, you need only inform the webhook listener that something changed in a given record. Then it’s up to the webhook listener to suss out the changes that cause its CDN to capture and host the content.

If by “scale” you mean a hundred thousand attachments all maintained by a single automation, this is possible because I’ve done it.