I am trying to use Airtable for my VueJS app’s database. I have it set to retrieve data using Axios. Everything renders find for the client, but I am wanting to have my site index on Google. I tested it using Google’s Mobile Friendly Test (search(dot)google(dot)com/test/mobile-friendly to see how the content and meta tags would render, but I receive an error message. This is for Googlebot.
See my site’s rendered HTML and show the content that shows when a normal user visits.
Error: X page resources couldn’t be loaded
Status: Googlebot blocked by robots.txt
It links me here (api(dot)airtable(dot)com/robots.txt) where i can see:
What I’ve Tried
Updating my apps robots.txt to add the below:
# Group 1
# Group 2
# Group 3
I tried adding a Crawl-delay for Googlebot in my Robots.txt
Screamining into the void (surprised this did not work)
Any help would be appreciated! :grinning_face_with_smiling_eyes:
Edit: The site is hosted on Firebase Hosting on a custom domain
Describe the climate where you are hosting this site.
Also, tell me how your site would behave if 100 users all made requests at about the same time? Would your Axios API calls open new connections for each of the 100 users? Or, would it cache data from Airtable and perform GETs based only on changed data in Airtable?
To be clear, a page request by a new user results is just one data request to Airtable? Literally one (and only one) call to the Airtable API?
Think about what a crawler is designed to do.
Imagine you have a catalog of items and a high-level query that exposes all the possible catalog items. For discussion purposes, imagine that each item is a separate page, and there are 500 items.
Google will quickly finds the high-level list of products and then attempts to index all 500 items. This results in a flurry of requests to your backend - a backend designed for the opposite of “flurries”. This is when Googlebot comes face-to-face with Airtable’s max API quota of five requests per second.
Your issue is not likely related to Google’s inability to crawl your content. Rather, it’s the opposite - it is not paced slowly enough to index all of the pages whose dependencies rely on a backend that is simply not designed to support indexation.
Your design should be modified to cache-forward Airtable data into a platform more capable of serving content (like Firestore).