Automatically get files from (protected) URLs using Sripting

[Jonas_Kockx]

Hey
I wanted to automate the process of fetching files from URLs and putting them in a Attachment filed. I found a script to do so at:

Setting Attachment URL dynamically Ask the community

Hello there, newish to AT, i wonder is there a way to set the URL of the attachment field dynamically using code ? i have used the formula field to build a URL to an image, and i know wish to set the URL of the attachment field to that URL ? doable ? thanks in advance for any help you can offer. Kevin

However, the URLs I want to fetch require being logged in into Webflow. So I was wondering if someone knew a method of making sure the URLs are fetching ‘as it was working in your own browser’. In my browser I can simply copy the URLs as I’m automatically logged in to that website.
Kind regards
Jonas

[Olpy_Acaflo]

Bill.French:

What matters is what URL will you hand off to Airtable for loading an attachment. If the API retrieves a secure URL and hands it off to Airtable, the attachment process will fail because Airtable itself requires ALL attachments to be publicly accessible.

Yes @Bill.French and @ScottWorld,
this is the main problem to solve anyway !

Many Thanks,

olπ

[ScottWorld]

@Bill.French @Olpy_Acaflo

Ahhh, I see, that’s a good question! And something I overlooked above. It definitely depends on the API. For example, Google Drive has the option to allow access to a private URL, as long as you know the private URL.

I’m not familiar with WebFlow, so if it doesn’t operate in a similar fashion, then some intermediate steps would be needed in Integromat to temporarily transfer that file into a publicly-accessible space (such as Google Drive) where it can temporarily gain a public URL.

So, an Integromat scenario could look like this:

Authenticate with WebFlow > download the file to Google Drive > send the Google Drive URL to Airtable > delete the file from Google Drive.

[Bill_French]

ScottWorld:

Ahhh, I see, that’s a good question!

LOL! Finally, I got through. :slightly_smiling_face:

ScottWorld:

It definitely depends on the API.

Actually, it depends on the visibility options of “private” URLs in the target document platform.

ScottWorld:

Google Drive has the option to allow access to a private URL, as long as you know the private URL.

Ergo, it’s not really private nor is it secure (at least not in the sense or spirit of the target document platform). It is private and secure if your definition includes links to documents that are openly accessible by anyone in possession of said link. :winking_face:

ScottWorld:

some intermediate steps would be needed in Integromat to temporarily transfer that file into a publicly-accessible space (such as Google Drive) where it can temporarily gain a public URL.

Ergo - you need a proxy. But this definition of a proxy likely violates the spirit of why the documents are not available as openly accessible URLs in the first place. Perhaps this is not really a requirement, but we have to assume it is because documents are typically secured for good reason.

ScottWorld:

Authenticate with WebFlow > download the file to Google Drive > send the Google Drive URL to Airtable > delete the file from Google Drive.

Your suggested process is missing a very important step -

… it must also set the security settings for the downloaded file in Google to “Accessible to anyone with the link…”.

What you’ve described here is similar to what I described in my first response. Can Integromat perform all these steps? Perhaps it can. I’ve been fortunate enough to get hired many times to create proxies like this because it’s very difficult to achieve this process without precise timing in a no-code environment.

One thing to remember - any Google Drive document with the security setting “Accessible to anyone with the link…” is also discoverable by anyone in the G-Suite domain for which the account exists. And by “discoverable” I mean searchable, viewable, copyable - it (by definition) is a wide-open document to everyone in the domain even if they do not possess the URL.

To be clear, in my approach, I do not instantiate “intermediate” temporary documents in Google Drive that are exposed publicly or at the domain because it increases the security attack surfaces by a factor of 3 and also broadly publishes the document to the organization’s users.

The most secure way to do this while satisfying the spirit of the secure document URLs is to established a signed URL that is accessible only through a webhook in Google Apps Script. This makes it possible to host (for a few seconds) a document that can be retrieved, but which doesn’t actually exist in Google Drive.

[Bill_French]

ScottWorld:

As far as I know, using Airtable’s scripting app reveals your API key to all of your users, even read-only users.

I don’t think this is the case any longer. Indeed, use of any third-party API in a script requires that script has knowledge of a security context of some type. However, isn’t this how you defend against overtly making API tokens and other login credentials difficult to view?

New beta: script settings Scripting app

Hi everyone! We’re excited to announce a new beta feature for the scripting block: script settings! With script settings, you can define settings for your script: a title, description, and settings items (table, field, view, text, number, and select). From your definition, we render a settings UI. Users of your script can update the settings and run the script without needing to modify the code. You can sign up for the beta via this form. You’ll only be able to use script settings if you’ve sig…

[ScottWorld]

Bill.French:

Can Integromat perform all these steps? Perhaps it can.

Yes, Integromat has the ability to change permissions of Google Drive files.

[Bill_French]

ScottWorld:

Yes, Integromat has the ability to change permissions of Google Drive files.

Cool. And can it reverse the permissions and/or remove the file when (and only when) Airtable’s attachment process finishes?

[ScottWorld]

Bill.French:

Cool. And can it reverse the permissions and/or remove the file when (and only when) Airtable’s attachment process finishes?

Yes, it can do both things.

[Bill_French]

Excellent! Someone should publish that use case because it’s very costly to script.

[ScottWorld]

Actually, even better, Integromat can temporarily download files into its own cache while running a scenario… so you could eliminate Google Drive altogether and just download the attachment into Integromat, upload it to Airtable, and it will be instantly deleted when the scenario is finished.