Diagnose script interactions

I am trying to figure out an issue where attachments in my Airtable records are being deleted - allegedly by me according to the record history - when I am doing nothing of the sort.

I am using Integromat to create an Airtable record from incoming emails - the script includes populating an Attachments field in Airtable with certain email attachments.

I then have a native Airtable automation that classifies these incoming email message records based on their content and updates a Message_type field

That all works just fine, however in many, but strangely not all cases, the attachment then mysteriously gets deleted…

Here’s the record history pane from one of these records (I’ve obscured the data fields)

The Activity history shows the initial creation of the record via API, including an attachment, from Integromat, then the classification step using my Airtable script - in this case is a ‘no match’. All this script does is set a value in a ‘Message type’ field based on sting matching against content in the email body

But then the Activity history it shows the attachment is deleted - by me! But I did not delete it!

How can if work out what is actually driving this unwanted deletion of the attachment?

You’re running into an issue that sometimes happens when trying to add attachments via Integromat. If you search the forum, you’ll likely find other threads on this subject, but long story short, the URL of the attachment you’re adding needs to be a permanent place where an actual document lives online. If the Integromat module you’re using provides a URL for a file that’s served up on the fly, that will often lead to the situation you’re encountering: the file is added, then it disappears moments later.

Could you share more details about the modules you’re using in Integromat? We can probably provide more specific guidance based on that.

Hi Justin, I think you actually helped me with the classifier Airtable automation I’m now using, so I guess this is now an example of no good deed going unpunished… Sorry about that, and thanks for all your help :grinning:

The relevant piece of my Integromat flow looks like this:

I’m taking file attachments from incoming emails, filtering for PDFs only, saving them to Gdrive, then adding them into an Attachments field in an Airtable record. I don’t actually need the files in Gdrive at all, but this was the only way I could figure out how to handle the scenario where there are multiple file attachments

Since its Gdrive, there is a stable URL where these documents do actually permanently live, and since this does work some of the time, I figure this cant be a permissions issue on the Gdrive…

A potentially relevant bit of info is that this behaviour seems to be inconsistent - the attachment deletion seems to happen a bit more than 50% of the time, but not every time.

Are you possibly mistaking a deletion with cases that the attachment seemed to have occurred (should have occurred) but wasn’t actually sustained?]

This is a well known issue that lots of people have experienced, but this would also be the first known case where Integromat was involved.

Not from my experience. I ran into this a while ago when I was using Integromat more often. Only recently did I figure out the right combination of modules to make it work reliably. It’s a minor pain, but it does the job. I’m tied up at the moment, but I’ll post an update later with details on the solution I found.

Good to know. For more than a year this error condition has been discussed in various threads and one common element has been that integration systems like Zapier and Integromat never exhibited these issues. Even @ScottWorld was quick to point out that Integromat suffered no such problems.

If Integromat can be used to demonstrate this problem in a repeatable way, it may be explained if they chose to update their platform to use the REST API instead of the original partner-developed code supported through Airtable prior to the REST API being made available.

Looking forward to your next post.

I’d love to see the workflow that @Justin_Barrett created for this issue!

I haven’t personally experienced this issue myself, but I wonder if the attachment hasn’t been fully uploaded to Google Drive before you try to insert the attachment into Airtable, especially if it’s a larger attachment? Maybe try adding a “Sleep” module after the Google Drive module to pause the workflow for a few seconds?

That’s just my guess, but I’m not sure.

I fully tested this hypothesis and many times. Google Drive will not return a URL until the document has been fully instantiated. Ergo, it’s impossible to have a URL to a document before the document has been fully saved to Drive.

This approach has been tested 60,000 times by me; it has no bearing on the success rate. We’ve also tested a framework that delays and retries up to ten times with no known impact on success rates.

All of my tests suggest there is a failure occurring in the Airtable platform when attachments are introduced from the public API.

Since this issue has never presented itself in Zapier (or Integromat as far as I know), I can only assume that neither of these platforms utilize the public [REST] API that we all must use.

This is great information to know! Thank you!!

Very interesting.

Sorry for the delay. Here’s the relevant part of my scenario (the full thing is the mess I was remembering earlier; this part is actually pretty simple):

Even though the “Upload a File” module does provide a sharing URL for the uploaded file, it’s not persistent enough for Airtable’s taste.

I think that Google has been updating its Integromat modules, because I don’t recall seeing the “Get a Share Link” module when I ran my earliest tests last year. I could be wrong. However, when I built this test scenario a little over a month ago, I discovered it, tested it, and subsequently rejoiced. The link that it provides is sufficiently persistent, and the file doesn’t vanish from Airtable shortly after arriving like a trick-or-treater through a convenient trap door that’s triggered by the doorbell…wait, what was I talking about?

Here’s the setup for “Get a Share Link” if it helps.

In the Airtable module, the attachment field setup looks like this (I’m using my own custom filename created in Airtable, but you could probably use the native file from Google if you want):

@Richard_Foxworthy In this case, my gut says that the “Get a Share Link” module would need to go inside the aggregation structure. I haven’t used Integromat aggregators yet, but I’m guessing your current setup aggregates share URLs from the “Upload a File” step. Switch it to pull the share URLs from the “Get a Share Link” module instead.

Hi @Justin_Barrett and @Richard_Foxworthy

You are right Justin, Google did an upgrade with GDrive and you now need the first two modules, in tandem, to work.

Hope this might help!

Mary K

Thanks @Justin_Barrett and all, I will apply that second Gmail step which hopefully will work.

I’ve found a further variation on this that may be related - another of my file attachment scenarios also displayed this behaviour - guess I’m just lucky - but the Integromat scenario is different.

In this second case, I’m parsing a file download link from the email message body, then inserting the link into an Airtable Attachments field. This works often enough, but also fails regularly.

Airtable support had this to offer:

That activity in your revision history typically means Airtable was unable to process the attachment. So although it looks like it was uploaded and then deleted, it’s actually the case that we never had the full file. This could be due to the URL not being a publicly available URL (e.g., it requires a login to access), or the Integromat integration is sending URLs in a format we can’t parse.

Would you mind confirming that the URLs you’re sending are upload-able using the UI when you access the base itself? For example, if you click the “+” in an attachment field and select the the URL option below, does the file upload successfully that way?

And sure enough my example URL failed in the Airtable upload tool, even though it succeeds in my browser. I’ll re-engineer my Integromat flows to include an explicit save-file-to-GDrive step followed by the Get Share Link that @Justin_Barrett identified - -hopefully that will cure both my problems!

thanks again everyone for help on this.

2 Likes

Unfortunately this additional Gdrive step has proven not to solve my issue.

I’m now using an Integromat flow like this:

I have today found an example where the generated Gdrive Sharelink URL does work in the Airtable manual file upload tool, but it fails in the same way as before when the same URL is used from Integromat.

I’m using the ‘Web Content Link’ URL from the Integromat ‘get a share link’ module - this is a Gdrive URL with &export=download appended to the URL.

Which seems to suggest that this issue may actually be related.

following up on this @Justin_Barrett i am seeing the exact same thing here in spite of the sharing script.

this seems to work better in smaller batches than larger ones…

funnily, this happens JUST the same way with airtable’s own automations - once you pass this in bulk. I followed the approach here Native URL to Attachment which works, also with small arrays of attachments and does just fine in testing. But then, fails when triggered in large batches. @Justin_Barrett sorry for spamming but this is just data getting lost without as much as a confirmation

@Martin_Kranz I’m not sure how to address this. In my case, I was just working with individual files, so perhaps the issue only occurs with bigger batches. I suggest reaching out to Airtable support directly to make them aware of this and see if they can help. I’m just a fellow user, and at this point I’m out of ideas without delving deeper into further testing (which I’m afraid that I don’t have time to do at the moment).

I think we’re all chasing a ghost; an inherently unreliable and inconsistent upload feature that fails from time-to-time and sometimes involving many attachment attempts and sometimes for just a few attachment attempts.

It’s also likely there is at least one workaround and perhaps @Justin_Barrett has found one (or so he may think he has until he discovered he hasn’t). :wink:

23 days ago and 27 days ago I suggested this issue may be related to this well-documented collection of observations which are still being treated by Airtable as nothing more than typical user-related failures - i.e., pilot error. Responses like this from Airtable’s support group exhibit insensitivity to the deep details and tests the community has conducted across many similar threads.

From Airtable Support:
“So although it looks like it was uploaded and then deleted, it’s actually the case that we never had the full file. This could be due to the URL not being a publicly available URL (e.g., it requires a login to access), or the Integromat integration is sending URLs in a format we can’t parse.”

No person who has actually read all of the issues related to attachment uploads would respond in this fashion because nearly every thread includes validated tests where typical missteps have been accounted for and eliminated to demonstrate attachment failures.

This is like a RICO case; it must be prosecuted as a whole, not as individual support tickets.

Where is the product manager who can own this calamity and get it resolved?

3 Likes

I did actually read most of the documentation online - definitely public url, as the same url uploads just fine via the file upload in the UI, so this can’t be it, really. It’s definitely on the airtable side, as it happens the same way using their own automations.

I am also in touch with support about this, who respond the same way so it’s not looking like they have any knowledge to address this either. Their response sounds a LOT like what @Bill.French has pasted here.

PS apologies, i was under the impression you’re community moderator @Justin_Barrett

just for the record, i am attaching a screen recording of Justin’s scenario, mildly adapted to work with multiple upload urls per record they should be uploaded to, and how this fails to keep the files, even though in some cases names, mime type etc appear to be known to airtable to the extend of displaying this info in the ux.