Automations are not working properly anymore

[andrei_andronic]

Hello,

We have a built a complex database for a processing facility, over the past 10 months. As of today, we have about 35 automations running (at various frequencies) and the main base will soon reach 100k entities out of 125k.

Until Friday, there haven't been any slowdowns or any load issues. There are 3-4 automations that are running once every few minutes especially during 9am - 6pm. We have a steady 10k entities/month increase rate. And no development has happened since Friday.

On Saturday it was reported that the platform started to slow down and entities generated through automations are starting to take minutes to appear.

This morning, the automations runs are no longer taking <1 minute to finalize, we are looking at over 30 minutes runs. The queue of runs is chaotic, runs that started later are completed sooner that previous runs and the entire base is full of errors because of this.

The company is facing serious penalties due to high processing times.

The support email we received it's disappointing. We've been told that "automations stuck in pending status are typically a result of overall base performance/load.". For one, there haven't been any changes on our behalf that would dramatically increase the frequency of the automation runs. Secondly, we are well under the 125k entities limit, so any slowdowns are not acceptable at this stage.

We seriously considered upgrading to a Enterprise plan in 2-3 months but this situations makes us seriously considering canceling this plan.

Did anybody else faced similar issues on a ~100k entity database?

Thanks!

[Snoochy_boochie]

Running scripts in automation has noticeably slowed down on my end as well - 5-10 minutes for one simple script.

At first I thought it was not working at all... however, it seems it may be working, just VERY slowly.

Normally this works very quickly, within a few seconds.

Something seems to be glitching.. I've contacted support to let them know.

[andrei_andronic]

It looks like very similar to what we faced today!

But for us it's also important that the run order was entirely mixed up! We need to record inbound and outbound transports in the platform and prepare documents for them. Each document has an unique number. But because the run order was not working properly, then:

• some transports received duplicated documents with different number
• some newer transports received their documents a lot sooner than older transports even though the automation was identical
• some transports were duplicated entirely

In the end we had no choice than to delete some of the older entities and we went from ~100k to ~80k. At a first glance, the performance improved but I'm waiting on a confirmation from the team.

But this is not a solution we are ready to accept considering how much $ the client pays for subscription + paid for development.

[Snoochy_boochie]

Upon further investigation, it looks like this isn't just an issue with the scripting feature... I adapted it to just use a normal conditional action (update record) and it's still going just as slowly (questionable if it will work at all)

I'm not working with tens of thousands of records here - in this case, a total of less than 1,000, with less than 100 being called to perform an action on

This is the first time I've experienced this... hopefully Airtable can fix!

[andrei_andronic]

It doesn't look like deleting ~20k entities generated any improvement.

Here's a history of just 1 run that creates another entity (from which we generate a PDF preview with a plugin), today:

• Run started at 0858, finished at 0858 - <1 minute runtime
• Run started at 0927, finished at 0927 - <1 minute runtime
• Run started at 1100, finished at 1105 - 5 minutes runtime!
• Run started at 1103, finished at 1156 - 53 minutes runtime!
• Run started at 1138, finished at 1156 - 18 minutes runtime!
• Run started at 1210, finished at 1215 - 5 minutes runtime!
• Run started at 1214, finished at 1235 - 21 minutes runtime!
• Run started at 1255, finished at 1306 - 11 minutes runtime!

This is just a snippet of one single automation run out of ~10 that were not working properly today.

[msulivan29]

Ours are running exceptionally slow today as well. Simple automations such as one where we click a checkbox to update a record are taking minutes to execute (normally runs within 5-10s at most). 

I've contacted support as well. Hopefully they can get this resolved quickly.

[isaiahmiller97]

This has to be an Airtable issue today yet their airtable status is not very forthcoming.

I emailed our Enterprise support, but this is totally unacceptable.  Our company is at a total standstill and we are not able to carry out basic business operations due to this, yet Airtable is technically showing as 'operational'.

This is an issue that is just occurring today.  They had a similar issue for days and more prevalent back in June of last year.  Haven't had issues since then until now.

[Arjen]

Same problem here, automations that normally take seconds now takes minutes and then gives errors (never had that). There is one automation that copies a record from one tab to antoher. Now it instead of copying it, it throws an error and starts duplication everything for no reason.... 

[andrei_andronic]

It looks like everyone involved here encountered the same issue over the past few days.

We got a mail from Airtable Support 2 days confirming that a fix was applied and everything is fine. We could confirm that indeed, until a few minutes ago, the automations' run time returned to normal.

But now everything moves slowly again. We are staring again at automations that are stuck over 5 minutes at each step!

What is happening?

[andrei_andronic]

At least now we receive an error message.