Skip to main content

I have the following use case: I have images saved as attachments within a table. I need to automatically upload any newly created images via API (not Make or Zapier due to volume) to an 'Input' bucket on S3. From here the images are manipulated by another API and saved to an output folder. When a file is uploaded to the output folder we would send this back to airtable.

Can someone give me some general direction in writing a script to enable this? I understand I will need a presigned URL from AWS which is well documented, but I'm struggling on how I might do this using Airtables API. Any help very much appreciated!

Many thanks

Edward

Does your S3 bucket allow anonymous uploads? If not, I'm afraid that authenticating into AWS using a script is going to be very, very challenging. After looking at the AWS S3 documentation, I decided to use Make.com for our AWS S3 uploads. Downloads from S3 to Airtable on the other hand were fairly simply. We setup our files on AWS to allow public downloads.

Out of curiosity, what is the other API that is manipulating the images? Could you send the initial file to that API directly, and skip the AWS upload?

Also out of curiosity, what type of volume are you trying to process (number of images and size of images)?

Does your S3 bucket allow anonymous uploads? If not, I'm afraid that authenticating into AWS using a script is going to be very, very challenging. After looking at the AWS S3 documentation, I decided to use Make.com for our AWS S3 uploads. Downloads from S3 to Airtable on the other hand were fairly simply. We setup our files on AWS to allow public downloads.

Out of curiosity, what is the other API that is manipulating the images? Could you send the initial file to that API directly, and skip the AWS upload?

Also out of curiosity, what type of volume are you trying to process (number of images and size of images)?


Hey Kuovonne, thanks very much for your response. So to elaborate a little more on the use case...the end goal here is to create an almost fully automated content for our poster businesses. The first part of this involves automatically downloading jpeg images of various poster designs into airtable from our server. This part is done. What we then need to do is create mockups of these designs in photoshop. This is where the upload to s3 comes in. We are looking to utilise photoshop's API to insert our designs into a smart object within a series of PSDs (even running manual scripts on a local machine is time consuming and clunky at scale). We're not 100% sure of volume but could easily be 50 or so mockups per day though will be less initially.

Because adobe doesn't currently offer cloud storage they require input/output to be done via pre-signed URLs (for AWS that is). So no we couldn't bypass this part unfortunately.

Another difficulty is that the max expiry on a presigned URL is 7 days so I'd also have to work out how to programmatically create these URLS to update within airtable. However, for now I'd just take working out how to get an image from airtable to s3 with a pre-signed URL within the 7 day expiry as a start!

Any help is enormously appreciated!

Uploading 50 images per day sounds like a volume that Make.com could easily handle. What am I missing?

Doesn’t Adobe Creative Cloud provide storage?

Will public urls work instead of presigned urls? It sounds like you have a lot of moving parts that all need to fit together for your system go work. 

Uploading 50 images per day sounds like a volume that Make.com could easily handle. What am I missing?

Doesn’t Adobe Creative Cloud provide storage?

Will public urls work instead of presigned urls? It sounds like you have a lot of moving parts that all need to fit together for your system go work. 


Hi Kuovonne, re Adobe Creative Cloud, yes they do offer storage but I am integrating with the photoshop API which does not - FYI: https://developer.adobe.com/photoshop/photoshop-api-docs/general-workflow/

And yes, I guess we can go with Make just to get files into s3, though would prefer to connect directly. I'll post the solution once I've worked it out. Despite the complexity (for me at least!) i'd imagine its something that people might have a need for (there are numerous other actions you can perform with the PS API)

Hi Kuovonne, re Adobe Creative Cloud, yes they do offer storage but I am integrating with the photoshop API which does not - FYI: https://developer.adobe.com/photoshop/photoshop-api-docs/general-workflow/

And yes, I guess we can go with Make just to get files into s3, though would prefer to connect directly. I'll post the solution once I've worked it out. Despite the complexity (for me at least!) i'd imagine its something that people might have a need for (there are numerous other actions you can perform with the PS API)


If you are able to connect directly, please post back with your findings! Best of luck to you!

If you are able to connect directly, please post back with your findings! Best of luck to you!


Ok so I have an update on this. The original reason I wanted to upload to S3 was to perform a series of image manipulations via the photoshop api. So basically the flow is our server creates poster designs using a table of data in airtable.We fetch the images generated as attachments from our server via webhook and store in the same table. We eyeball the designs to check they look ok and then (and this is the missing part I need to connect it all) send to an s3 bucket via AWS api gateway as a put request. This triggers a lambda function which runs the code via the photoshop api and saves the mock up images to an output folder. We then use imagekit.io to serve the images at various sizes and aspect ratios for our content marketing. So, this has been VERY difficult to figure out and I feel it should now be fairly simple to send these image attachments to my aws api endpoint. Can anyone explain how I would create a script based on X happening in airtable which uploaded the files. For example, I can send the files using postman using the url and a put request. Thanks!

Does your S3 bucket allow anonymous uploads? If not, I'm afraid that authenticating into AWS using a script is going to be very, very challenging. After looking at the AWS S3 documentation, I decided to use Make.com for our AWS S3 uploads. Downloads from S3 to Airtable on the other hand were fairly simply. We setup our files on AWS to allow public downloads.

Out of curiosity, what is the other API that is manipulating the images? Could you send the initial file to that API directly, and skip the AWS upload?

Also out of curiosity, what type of volume are you trying to process (number of images and size of images)?


The solution I found was creating an AWS API gateway and allowing put requests to an S3 bucket. This was actually quite straightforward and well-documented on youtube. Permissions and roles are a little tricky but doable as a no/low coder and again well documented. However, there was an issue with uploading to a specific bucket and 'folder' which I got some help with. as explained below:

"The problem with trying to create an item in the S3 "input" folder is S3 doesn't really have folders in the underlying bucket, only objects. It simulates a folder in the GUI using the backslash (/) in the key name, but actually all it's doing is creating an object called "input/name.jpg", rather than an object called "name.jpg" in the "input" folder. The other issue is the API gateway uses the same backslash (/) to split a path up. So when you try to put an object with the key including the backslash, the API gateway sees it as a new path, and you get the Missing Token error as that path doesn't exist. It took me a little while to figure out the solution, but the trick is to use the URL encoded form of the backslash in the URL which %2f i.e. "input**/name.jpg" becomes "input%2f**name.jpg".

This allows me to put image attachments to pbposters-images/input using https://my-gateway-my-region.amazonaws.com/prod/pbposters-images/input%2fnew.jpg

So I can now upload directly from any airtable base straight into S3 with the simple script below (created with help from chat GPT):

 

// Replace 'YOUR_API_GATEWAY_URL' with the URL of your API Gateway endpoint
const apiGatewayUrl = 'https://gateway-region.amazonaws.com/prod/pbposters-images/input%2f';

// Replace 'YOUR_TABLE_NAME' and 'YOUR_FIELD_NAME' with the name of the table and field that contains the image attachment
const table = base.getTable('YOUR_TABLE_NAME');
const record = await input.recordAsync('Process PSDs', table);
const attachment = record.getCellValue('YOUR_FIELD_NAME')[0];
const attachmentName = record.getCellValue('fileName');
const imageUrl = attachment.url;
console.log(imageUrl);

// Download the image data from the URL
const imageData = await remoteFetchAsync(imageUrl, { method: 'GET' }).then(response => response.arrayBuffer());

// Set up the HTTP request options for the PUT method
const options = {
method: 'PUT',
headers: {'Content-Type': 'image/jpeg'},
body: imageData,
};

// Send the image data to the API Gateway endpoint using the PUT method
const response = await remoteFetchAsync(apiGatewayUrl + attachmentName, options);
console.log(response);

 

The solution I found was creating an AWS API gateway and allowing put requests to an S3 bucket. This was actually quite straightforward and well-documented on youtube. Permissions and roles are a little tricky but doable as a no/low coder and again well documented. However, there was an issue with uploading to a specific bucket and 'folder' which I got some help with. as explained below:

"The problem with trying to create an item in the S3 "input" folder is S3 doesn't really have folders in the underlying bucket, only objects. It simulates a folder in the GUI using the backslash (/) in the key name, but actually all it's doing is creating an object called "input/name.jpg", rather than an object called "name.jpg" in the "input" folder. The other issue is the API gateway uses the same backslash (/) to split a path up. So when you try to put an object with the key including the backslash, the API gateway sees it as a new path, and you get the Missing Token error as that path doesn't exist. It took me a little while to figure out the solution, but the trick is to use the URL encoded form of the backslash in the URL which %2f i.e. "input**/name.jpg" becomes "input%2f**name.jpg".

This allows me to put image attachments to pbposters-images/input using https://my-gateway-my-region.amazonaws.com/prod/pbposters-images/input%2fnew.jpg

So I can now upload directly from any airtable base straight into S3 with the simple script below (created with help from chat GPT):

 

// Replace 'YOUR_API_GATEWAY_URL' with the URL of your API Gateway endpoint
const apiGatewayUrl = 'https://gateway-region.amazonaws.com/prod/pbposters-images/input%2f';

// Replace 'YOUR_TABLE_NAME' and 'YOUR_FIELD_NAME' with the name of the table and field that contains the image attachment
const table = base.getTable('YOUR_TABLE_NAME');
const record = await input.recordAsync('Process PSDs', table);
const attachment = record.getCellValue('YOUR_FIELD_NAME')[0];
const attachmentName = record.getCellValue('fileName');
const imageUrl = attachment.url;
console.log(imageUrl);

// Download the image data from the URL
const imageData = await remoteFetchAsync(imageUrl, { method: 'GET' }).then(response => response.arrayBuffer());

// Set up the HTTP request options for the PUT method
const options = {
method: 'PUT',
headers: {'Content-Type': 'image/jpeg'},
body: imageData,
};

// Send the image data to the API Gateway endpoint using the PUT method
const response = await remoteFetchAsync(apiGatewayUrl + attachmentName, options);
console.log(response);

 


Hi @Edward1976,

Would you know how to replace the Attachment URL with the Expiring Download URL in your script code?

Hi @Edward1976,

Would you know how to replace the Attachment URL with the Expiring Download URL in your script code?

 

Hi @Edward1976,

Would you know how to replace the Attachment URL with the Expiring Download URL in your script code?


@siliceous Looks like we're trying to accomplish the exact same thing!

Did you ever find a solution that worked for you?

I'm trying to do this with PDFs in an Airtable automation.

Thanks in advance!

@siliceous Looks like we're trying to accomplish the exact same thing!

Did you ever find a solution that worked for you?

I'm trying to do this with PDFs in an Airtable automation.

Thanks in advance!


HI @goodthings 

Sorry about the delayed reply. I've been flat out.

We decided to use a different method of moving Airtable attachments around that works for a particular use case. What is the workflow you would like to achieve?

Regards.

HI @goodthings 

Sorry about the delayed reply. I've been flat out.

We decided to use a different method of moving Airtable attachments around that works for a particular use case. What is the workflow you would like to achieve?

Regards.


Hey @siliceous thanks for the response. I've been out for the US holiday so sorry for the delay as well.

We're sending documents (PDFs, spreadsheets) to S3 via Airtable. We want to store them on s3. That's it!

So far, I believe @Edward1976 above is the best without using a 3rd party tool like pipedream.com to handle it for you. Zapier and make.com don't have the native ability to download the file from Airtable's expiring file URL and stream it somewhere.

But if you have another way to do it- maybe with Javascript in Airtable's automations code editor, would love to hear!

Hey @siliceous thanks for the response. I've been out for the US holiday so sorry for the delay as well.

We're sending documents (PDFs, spreadsheets) to S3 via Airtable. We want to store them on s3. That's it!

So far, I believe @Edward1976 above is the best without using a 3rd party tool like pipedream.com to handle it for you. Zapier and make.com don't have the native ability to download the file from Airtable's expiring file URL and stream it somewhere.

But if you have another way to do it- maybe with Javascript in Airtable's automations code editor, would love to hear!


Hi @goodthings 

I hope you didn't have to wait too long for your holidays to come.

Zapier and make.com don't have the native ability to download the file from Airtable's expiring file URL and stream it somewhere.

Nothing is impossible.

Unfortunately, we are not at liberty to share our methods for the particular use case on a public forum.

All the best.

Does your S3 bucket allow anonymous uploads? If not, I'm afraid that authenticating into AWS using a script is going to be very, very challenging. After looking at the AWS S3 documentation, I decided to use Make.com for our AWS S3 uploads. Downloads from S3 to Airtable on the other hand were fairly simply. We setup our files on AWS to allow public downloads.

Out of curiosity, what is the other API that is manipulating the images? Could you send the initial file to that API directly, and skip the AWS upload?

Also out of curiosity, what type of volume are you trying to process (number of images and size of images)?

The solution I found was creating an AWS API gateway and allowing put requests to an S3 bucket. This was actually quite straightforward and well-documented on youtube. Permissions and roles are a little tricky but doable as a no/low coder and again well documented. However, there was an issue with uploading to a specific bucket and 'folder' which I got some help with. as explained below:

"The problem with trying to create an item in the S3 "input" folder is S3 doesn't really have folders in the underlying bucket, only objects. It simulates a folder in the GUI using the backslash (/) in the key name, but actually all it's doing is creating an object called "input/name.jpg", rather than an object called "name.jpg" in the "input" folder. The other issue is the API gateway uses the same backslash (/) to split a path up. So when you try to put an object with the key including the backslash, the API gateway sees it as a new path, and you get the Missing Token error as that path doesn't exist. It took me a little while to figure out the solution, but the trick is to use the URL encoded form of the backslash in the URL which %2f i.e. "input**/name.jpg" becomes "input%2f**name.jpg".

This allows me to put image attachments to pbposters-images/input using https://my-gateway-my-region.amazonaws.com/prod/pbposters-images/input%2fnew.jpg

So I can now upload directly from any airtable base straight into S3 with the simple script below (created with help from chat GPT):

 

// Replace 'YOUR_API_GATEWAY_URL' with the URL of your API Gateway endpoint const apiGatewayUrl = 'https://gateway-region.amazonaws.com/prod/pbposters-images/input%2f'; // Replace 'YOUR_TABLE_NAME' and 'YOUR_FIELD_NAME' with the name of the table and field that contains the image attachment const table = base.getTable('YOUR_TABLE_NAME'); const record = await input.recordAsync('Process PSDs', table); const attachment = record.getCellValue('YOUR_FIELD_NAME')_0]; const attachmentName = record.getCellValue('fileName'); const imageUrl = attachment.url; console.log(imageUrl); // Download the image data from the URL const imageData = await remoteFetchAsync(imageUrl, { method: 'GET' }).then(response => response.arrayBuffer()); // Set up the HTTP request options for the PUT method const options = { method: 'PUT', headers: {'Content-Type': 'image/jpeg'}, body: imageData, }; // Send the image data to the API Gateway endpoint using the PUT method const response = await remoteFetchAsync(apiGatewayUrl + attachmentName, options); console.log(response);

 

Can you please give the link to that youtube video which you said has this issue well-documented?

Reply


Terms & Conditions