Jan 15, 2024 05:53 AM - edited Jan 15, 2024 05:54 AM
Hello,
We have a built a complex database for a processing facility, over the past 10 months. As of today, we have about 35 automations running (at various frequencies) and the main base will soon reach 100k entities out of 125k.
Until Friday, there haven't been any slowdowns or any load issues. There are 3-4 automations that are running once every few minutes especially during 9am - 6pm. We have a steady 10k entities/month increase rate. And no development has happened since Friday.
On Saturday it was reported that the platform started to slow down and entities generated through automations are starting to take minutes to appear.
This morning, the automations runs are no longer taking <1 minute to finalize, we are looking at over 30 minutes runs. The queue of runs is chaotic, runs that started later are completed sooner that previous runs and the entire base is full of errors because of this.
The company is facing serious penalties due to high processing times.
The support email we received it's disappointing. We've been told that "automations stuck in pending status are typically a result of overall base performance/load.". For one, there haven't been any changes on our behalf that would dramatically increase the frequency of the automation runs. Secondly, we are well under the 125k entities limit, so any slowdowns are not acceptable at this stage.
We seriously considered upgrading to a Enterprise plan in 2-3 months but this situations makes us seriously considering canceling this plan.
Did anybody else faced similar issues on a ~100k entity database?
Thanks!
Jan 17, 2024 09:44 AM
Yeah - same here... this is totally unacceptable.
Jan 18, 2024 10:25 AM
Hey folks,
Thanks for reaching out about this outage. We recognize how critical automations are to you and we apologize for the inconvenience.
These automations have been restored and are now running correctly, but please note any runs that were triggered during the system outage will not run and remain in a failed state.
If you were impacted by this outage, you should have received or will receive communication on next steps, and if you have any further questions, please file a support ticket here.