Repeatable AirTable File Deletion

[stephenheindel]

Hello,

I, like many here in the AirTable forums, have dealt with seemingly "random" file deletions. One of the issues with this bug is that it is often sporadic. However, I have recently refactored one of my cloud processes that processes data from a record, generates a data file (application/json type) and puts it back into the record. I have a consistent set of records that I can run this process on, and I can literally watch my process write in all of the files, and then anywhere from 1-20 seconds later I see the files deleted  realtime! I know that my code is doing nothing to actually delete the files, and you can even tell from the record history that something is awry because the original file write is by our "Software" user and it specifies the Users token, and then the deletion is also done by the "Software" user but no token is specified (see attached screenshot). I feel confident that I have no old API key based processes that would be doing this action. 

(As an aside about the attached screenshot: While it appears that there is a 10 minute delay between the insertion and deletion events, this is actually just AirTable history compression. I watched, in real time, the file get put in place, and then deleted less than 1 minute later)

I am posting this in the hope that someone at AirTable in engineering would be interested in evaluating our process and collecting more data from our specific table and process. I would hope with a repeatable demonstration that the underlying cause for this bug (which still seems a mystery to the broader AirTable community) could be found.

Some additional helpful information:
The process is a NodeJS 18 AWS Lambda but I can run it locally and generate the same result. 

We are using the latest NodeJS SDK (0.12.2)

Probably not relevant, but this is the AWS SDK version that our s3 module uses

@aws-sdk/client-s3: 3.468.0

@aws-sdk/lib-storage: 3.472.0

[ScottWorld · Accepted solution]

Yeah, what's weird about your revision history is that if the attachment was deleted through either an Airtable Automation or an external automation using an old API key, Airtable would specifically say in the revision history "via API" to let you know that it was an automated process that made the change. Your revision history makes it seem like a human logged in and deleted the file.

Question: Did you check to make sure that this particular file was fully downloaded into the base to begin with? The paper clip icon sometimes (but not always) indicates that the file never made it into the base. I wonder if the file never fully made it into the base, and then if Airtable is doing some sort of an automated process on its own to clean it up.

I'm not sure if anybody in these forums will be able to help you solve this issue, because Airtable employees don't typically read our posts here. It's primarily Airtable users helping other Airtable users here. You should probably open up a support ticket on this issue with support@airtable.com, but be sure to report back here with what they tell you!

[ScottWorld]

p.s. Also, the paper clip icon can indicate that a file was corrupted. Even a mislabeled file extension would flag the file as corrupted. For example, a PNG file with a mistaken JPG extension would be flagged as a corrupted file. If that's the case, I wonder if Airtable is auto-cleaning up corrupted files.

In any case, Airtable shouldn't be deleting files without your permission!

[kuovonne]

How are you writing the files back to Airtable? Are you sending the Airtable a publicly accessibld URL to the file? The behavior you are describing matches what I would expect if you do not send a publicly accessible URL to the file. Airtable immediately recognized that you want to attach a file, but it takes a few seconds to actually upload and attach the file. If the upload process fails, it appears that the attachment was deleted. However, in reality, the file was never fully attached to begin with.

If your cloud service generates the content of the file, but does not provide you with a publicly accessibls URL for the file, you can upload the file to another service that will accept the raw data and temporarily host the file at a publicly accessible URL. Then send that URL to Airtable.

[ScottWorld]

Yep, this is exactly what I was talking about with my question above. In 99.99% of the times where this problem has cropped up, it is a user error because the file never made it into the base (i.e. the file wasn't accessible at a publicly-accessible link). Meaning that Airtable tried to access the file, but was unable to access it.

[stephenheindel]

Well I'm ashamed to admit that it was the obvious solution that I missed, that my file wasn't publicly accessible. I was completing an large upgrade to our cloud processes which included migrating from AWS SDK v2 -> v3 and I messed up setting the ACL in the new API call.

I just hadn't before witnessed *live* the behavior of a non-public link in AirTable, and the fact that the resultant history behavior of
- file added
- file deleted by the same user with no PAT or API indication
matched times when this has sporadically happened with normally working clients, I thought I had stumbled across something important.

It does make me think that the times I get sporadic deletion may be due to how the underlying AWS client was handling the file upload. We noticed the issue more with our hardware clients which often have questionable internet, and if the upload process is two steps
- Upload file to AWS
- Set ACL on uploaded file
It does make sense that a delay in the ACL call or a failure of the call all together could be the underlying issue that makes the files "seem" to disappear. It makes me wonder if a slight delay on those clients would go a long way in resolving the underlying file deletion issue (although it is probably a <0.1% and maybe even <0.01% issue now)

[ScottWorld]

Glad you solved it! 🙂 Yeah, Airtable handles that issue in an extremely odd way. Airtable actually makes the problem MORE confusing by the way that they handle the issue.