Help

Automations are not working properly anymore

Topic Labels: Automations
4582 11
cancel
Showing results for 
Search instead for 
Did you mean: 
andrei_andronic
5 - Automation Enthusiast
5 - Automation Enthusiast

Hello,

We have a built a complex database for a processing facility, over the past 10 months. As of today, we have about 35 automations running (at various frequencies) and the main base will soon reach 100k entities out of 125k.

Until Friday, there haven't been any slowdowns or any load issues. There are 3-4 automations that are running once every few minutes especially during 9am - 6pm. We have a steady 10k entities/month increase rate. And no development has happened since Friday.

On Saturday it was reported that the platform started to slow down and entities generated through automations are starting to take minutes to appear.

This morning, the automations runs are no longer taking <1 minute to finalize, we are looking at over 30 minutes runs. The queue of runs is chaotic, runs that started later are completed sooner that previous runs and the entire base is full of errors because of this.

The company is facing serious penalties due to high processing times.

The support email we received it's disappointing. We've been told that "automations stuck in pending status are typically a result of overall base performance/load.". For one, there haven't been any changes on our behalf that would dramatically increase the frequency of the automation runs. Secondly, we are well under the 125k entities limit, so any slowdowns are not acceptable at this stage.

We seriously considered upgrading to a Enterprise plan in 2-3 months but this situations makes us seriously considering canceling this plan.

Did anybody else faced similar issues on a ~100k entity database?

Thanks!

11 Replies 11
isaiahmiller97
6 - Interface Innovator
6 - Interface Innovator

Yeah - same here... this is totally unacceptable.  

MaddieJ
Community Manager
Community Manager

Hey folks,

Thanks for reaching out about this outage. We recognize how critical automations are to you and we apologize for the inconvenience.

These automations have been restored and are now running correctly, but please note any runs that were triggered during the system outage will not run and remain in a failed state.

If you were impacted by this outage, you should have received or will receive communication on next steps, and if you have any further questions, please file a support ticket here.