Yes. But… and there are many (my $0.02).

• Cross-domain security issues.



• Legal issues (read the fine print in their ToS).



• Banning (they know precisely who/what is scraping and your actions could affect the ability for people inside the Airtable domain to access their linked in accounts).



• Ideally, use the Linkedin API (it requires some payment I think).



• Scale; Airtable script blocks are very fast with regard to fields and tables; very slow for all other stuff and crawlers need lots of processing cycles.



• I doubt there are any limits on HTTP requests to external web sites, but a really thorough crawler built using Script block may demonstrate why Airtable should discourage (or prohibit) doing this.

Finally - (in my opinion) no one should ever build a crawler in Script Block because it would be too brittle, your investment would be exposed to far too many ways it could or should be shut down, and it’s not a sustainable approach - better to use something more viable for the task.

Certainly, small numbers of fetches of content over HTTP(s) should not be a big deal and it might work really well until it doesn’t. :winking_face:

Hello @Aswanth_Selva_Pragat, @Bill.French,


I had started building a python WebCrawler App that would then populate an AirTable via AT public API.


This WebCrawler was to meet several attentions from me including those explained here by Bill.


I always preferred the API of a site to any other data collection technique for the reasons explained by Bill and a lot more other ones that I discovered while practicing experimental low-scale Web Crawling lab’s attempts under python.


Then Script-BLOCK started and with it, javascript came into my life, taking up my free time and attention next to my work.


But as a javascript student, I discovered that everything I had started in python web crawling experimental attempts could be done from javascript: it was even interesting to dig into subjects like “headless” “chromium” “chrome” and “puppeteer” and also " write your own chrome extensions".


This message is given as a good intention under condition that it had to become practice.


Although, I could share my Chrome / Chromium by js Control and Automation curated URLs List if it is interesting someone.


But my today’s priorities are now on AirTable Script- and Custom- BLOCK so Web Crawling Projects went asleep.


Cheers,

olπ

Thank you very much - @Bill.French, @Olpy_Acaflo for your detailed inputs.

All the points that you have put out totally makes sense.

We do have access to LinkedIn APIs and are planning to leverage that for the process. But, I don’t think the scripting block is the right place to build this at. Probably a separate instance that integrates with Airtable through Airtable API and then accesses LinkedIn APIs to pull info and put it back into Airtable through Airtable APIs would be the better option.

Thanks again! Appreciate it much

I built a functionality to collect data using the CSS selector functionality of the Web Clipper rather than the Scripting Block. This would be most useful for an individual recruiter or sales rep who can collect information from accounts as they’re browsing.

It may require using the dedupe block to clean up every once in a while but I didn’t find it to be too much a hassle when I was using Airtable every day in my sales job.

Here’s a write up I did on the Web Clipper: https://www.notion.so/Building-a-LinkedIn-Scrapper-on-Airtable-af052e46f3454d40909990d75fbdfde8

Hey Olpy, I am looking to hire someone to help scrape date from TikTok posts and feed it into AirTable. Is this something you can help with, or perhaps can help point me in the right direction. Thanks a lot!

Hi @Max_Bernstein ,

Sorry for the late reply but :

• I am my own Airtable Consultant to cover my personal needs but I have no operational experience with external clients;



• Airtable and Airtable Scripting SDKs are sideProjects for me: currently I’m busy with my main job (in which I use Airtable on my own);



• I have archived the project mentioned in my message before yours:
  
  
  I have prioritized other Scripts that keep me very busy when I have free time.

I hope that in the Airtable Community, you will find someone to hire to satisfy your request: excellent Experts are regularly involved in the most active topics of the moment.

oLπ