Grant Research App

Hey ​@Stephen_Juth,

That is pretty cool!
1. Do you think it wold be possible to handle pagination somehow?
2. Did you test out different sites for scraping? Did you find any pattern as to what can be easily scraped and what the agent has difficulties with?

Mike, Consultant @ Automatic Nation 
YouTube Channel 

Thanks for sharing ​@Stephen_Juth! The power of deploying AI at scale with Airtable is amazing.

We’ve implemented similar approaches for Grant Management Systems for both EDU and Nonprofits.

In a similar way, we developed a scraping workflows to extract publications from Google Scholar, analyze content and citations.

​@Mike_AutomaticN 

Thanks!

Here are my answers:

1. Yes, definitely doable, at least for Grants.gov. I tested this out with an AI action that has this prompt:
   
   “You are a web scraping agent. Visit the URL provided as input. From the search results table on the page, determine the max pagination value. Return only that number with no other text or symbols.
   
   URL input:

   https://simpler.grants.gov/search?utm_source=Grants.gov&query=<query goes here>.
   
   Example output: 5
   
   --
   After this, I used a code block that builds an array of integers from 1 to the value returned by the AI action. However, scraping a URL with a pageNum will require an iteration within an interation, which doesn’t look possible (never tried it until now). So, either the pageNum iterator can load another table with the URLs to be parsed individually, handled by my first “Find Opportunities” automation (per URL), or it could invoke that automation directly via a webhook.

2. Only Grants.gov for now. Actually, I did look at another similar site that did not include the keyword or page # in the URL, which makes it much harder. In that case, I would probably use Data Fetcher (which I love) with Apify for robust web scraping. But I’ll probably stick with sites that have URLs structured like Grants.gov, since it’s *really nice* to be able to do all of this with Airable AI and less techie solutions.

HTH!

Note: I realize point #1 might be a mouthful. So, feel free to let me know if you have any further questions.

That’s very helpful to know, ​@gaston. I appreciate you sharing this!

Interesting, thanks ​@Stephen_Juth!!

Mike, Consultant @ Automatic Nation 
YouTube Channel