Problems with Zapier / Google images as attachments

SAQA_Staff · ‎Jan 20, 2020

We have several Zaps that pull images from Google drive (put there via Formstack) and add them as attachments to a record. These zaps have been working perfectly for over a month, no problems. Since 01/10, half of the images aren’t being attached at all. This affects multiple forms / bases / zaps and is causing us a lot of problems as we have to attach the image files manually.

No errors are noted in Zapier and their techs haven’t seen anything odd. Haven’t had a reply from Airtable techs yet. I am slightly suspicious that this started happening after Airtable did a update to reduce load times. Has anyone else experienced problems since then? Or is it a google issue perhaps?

Bill_French · ‎Jan 21, 2020

I can confirm this is not a Google issue.

I have spent the past 12 days fully documenting and testing uploads from Google Drive that started failing 19-Dec-2019 and continue to fail roughly 30% of the time. The app in question was bullet-proof for three months and then this issue arose.

I crafted a number of metrics and even some additional processes to log failures and without question, I was able to rule out issues with Google Drive or latent permissions handling. This leaves me with great suspicion that the likely cause is Airtable.

To overcome this problem we believe we’re going to have to create a latent [API] process that constantly looks for failed uploads and attempts to retry. As you note, manually uploading seems to work fine which tends to rule out Google and the document itself or its permissions as the issue.

@alex (at Airtable) has been very responsive to this issue and also agrees that it could be some sort of unknown issue lurking in attachment processing. Last I heard, the Airtable team was looking deeper into this issue.

SAQA_Staff · ‎Jan 30, 2020

Slight update. We have been running a parallel test using images stored in AWS instead of google drive. Zero problems with attachments. Since this problem really started once we added more forms/integrations, I do suspect that it’s a permission issue with Google (although Formstack has no problems saving the images - it’s only Airtable that can’t retrieve them). It seems to work slightly better when we use a unique google account for each form but there are still failures. For a variety of reasons we can’t switch over to AWS and have to stay with google so I really hope a resolution is found soon.

Bill_French · ‎Jan 30, 2020

Yep - I have seen similar positive results with alternate document storage models. However, depending on how quickly the attachment upload request is performed after the document is created in the external object store is apparently a dependent factor on the success rate.

We have ruled out permissions issues in the Drive platform. Although, we have not ruled out the speed at which Google applies open permissions to newly-created documents in Google Drive. We believe there are varying latencies in permissions being applied as new documents are saved into fully and openly shared folders. This is likely caused by Google’s global CDNs and there’s nothing that Airtable can ever do to improve this. As such, the only reasonable way that Airtable can improve its interoperation with file storage systems is to realize these nuances exist and perform attachment retries intelligently over time.

Indeed; our findings as well. And this process was flawless in more than 7 client systems for almost a year predating 19-Dec-2017 when it started to fail. I believe Aitable pushed a new release right about that time.

SAQA_Staff · ‎Jan 30, 2020

Interesting. We have added a delay step (tried 2-10 minutes) in Zapier to delay the Airtable step but it didn’t seem to decrease the failure rate. Perhaps we will increase this delay to see if it has any effect.

Bill_French · ‎Feb 05, 2020

We’ve seen mixed results with 2 to 15 second delays; there may not be a correlation to time delay which clearly points the finger at Airtable as the likely cause of this issue.