Does your S3 bucket allow anonymous uploads? If not, I'm afraid that authenticating into AWS using a script is going to be very, very challenging. After looking at the AWS S3 documentation, I decided to use Make.com for our AWS S3 uploads. Downloads from S3 to Airtable on the other hand were fairly simply. We setup our files on AWS to allow public downloads.
Out of curiosity, what is the other API that is manipulating the images? Could you send the initial file to that API directly, and skip the AWS upload?
Also out of curiosity, what type of volume are you trying to process (number of images and size of images)?
The solution I found was creating an AWS API gateway and allowing put requests to an S3 bucket. This was actually quite straightforward and well-documented on youtube. Permissions and roles are a little tricky but doable as a no/low coder and again well documented. However, there was an issue with uploading to a specific bucket and 'folder' which I got some help with. as explained below:
"The problem with trying to create an item in the S3 "input" folder is S3 doesn't really have folders in the underlying bucket, only objects. It simulates a folder in the GUI using the backslash (/) in the key name, but actually all it's doing is creating an object called "input/name.jpg", rather than an object called "name.jpg" in the "input" folder. The other issue is the API gateway uses the same backslash (/) to split a path up. So when you try to put an object with the key including the backslash, the API gateway sees it as a new path, and you get the Missing Token error as that path doesn't exist. It took me a little while to figure out the solution, but the trick is to use the URL encoded form of the backslash in the URL which %2f i.e. "input**/name.jpg" becomes "input%2f**name.jpg".
This allows me to put image attachments to pbposters-images/input using https://my-gateway-my-region.amazonaws.com/prod/pbposters-images/input%2fnew.jpg
So I can now upload directly from any airtable base straight into S3 with the simple script below (created with help from chat GPT):
// Replace 'YOUR_API_GATEWAY_URL' with the URL of your API Gateway endpoint const apiGatewayUrl = 'https://gateway-region.amazonaws.com/prod/pbposters-images/input%2f'; // Replace 'YOUR_TABLE_NAME' and 'YOUR_FIELD_NAME' with the name of the table and field that contains the image attachment const table = base.getTable('YOUR_TABLE_NAME'); const record = await input.recordAsync('Process PSDs', table); const attachment = record.getCellValue('YOUR_FIELD_NAME')[0]; const attachmentName = record.getCellValue('fileName'); const imageUrl = attachment.url; console.log(imageUrl); // Download the image data from the URL const imageData = await remoteFetchAsync(imageUrl, { method: 'GET' }).then(response => response.arrayBuffer()); // Set up the HTTP request options for the PUT method const options = { method: 'PUT', headers: {'Content-Type': 'image/jpeg'}, body: imageData, }; // Send the image data to the API Gateway endpoint using the PUT method const response = await remoteFetchAsync(apiGatewayUrl + attachmentName, options); console.log(response);
