Help

Re: Grabbing html from a url in record

1210 0
cancel
Showing results for 
Search instead for 
Did you mean: 
Adam_Wolach
5 - Automation Enthusiast
5 - Automation Enthusiast

I’m going to be keeping track of pages of items on craigstlist, facebook marketplace, etc. These items will eventually expire, and I was hoping to have a script to load the url saved in the record and return the html so I could parse something like “this post has been deleted” or a similar css or html element.

I tried fetch, which seems to give an error because of CORS. I also tried remoteFetchAsync which will only return “function” no matter my settings. There are no APIs available for these sites, at least craigslist.

Here is the code that returns “function”. Any ideas appreciated thanks!

const getResults  = await remoteFetchAsync('https://tampa.craigslist.org/hil/cto/d/land-lakes-2016-chevrolet-cruze-limited/7337285063.html');
console.log(await getResults.text);
1 Reply 1

Welcome to the world of JavaScript development!

I’d suggest trying to jump the gun on some basic concepts like the fetch API because you’ll save yourself a lot of trouble in the long run.

First of all, pick a “real” code editor that you like and start writing your scripts in it. VSC is pretty popular and entirely free.

This will help you realize how naive your original approach was - fetches need babysitting, and this one hasn’t even gotten off the ground yet.

Lesson #2 - always start debugging from the bugging part. Your console.log method returns a “function”? That means you aren’t even invoking it, just logging its type. Or rather, in this instance, it seems you aren’t accessing the response correctly.

But first things first: functions in JS are invoked with parentheses.

Do not get wrapped into the sugary syntax of the latest ECMAScript specification that Safari will support in 2035.

Until you understand this by heart, try writing every function like this:

    function add (h,r}{return h + r};

Even if only to rewrite it immediately afterwards:

    const add = (h,r) => h+r;

I won’t cover the very latest use case because you already fell victim to it.

Namely, your console isn’t logging anything but the fact that response.text is a reference type belonging to a function. Same thing would happen if you’d try running, say, console.log(add) from the above example.

The good news is that the syntax will become second nature if you keep at it before long. What you want to be focusing on in the meantime and forever and ever is logical thinking and getting into the habit of doing something, recording what happened, recording what you were expecting to happen, then figuring out the cause of the discrepancy asap.

How you want to be writing these things in the beginning, overly expository reference names and all that:

async function fetchRemote (url,response,error){
    await remoteFetchAsync(url)
        .then(response=>{ 

          //handle the response here, response.status() is an ok start here

}).catch(err=>{
         //account for error cases here, as needed
              
          if(301) //just a random example of what you might want to do
           {console.log('hello, darkness, my void friend')}
       
      })
         };`
      }

I haven’t actually tried running this but it should be enough to get you going. Remember, get a real code editor, VSC, Glitch, JSFiddle, whatever, a good autocomplete functionality is paramount to almost every workflow, particularly while you’re still learning and especially if you’re only at the beginning.

Some resources you might find helpful:

JavaScript for Cats - this is pretty amazing and I’m not even a cat.

HTTP response status codes - HTTP | MDN (mozilla.org)

Beast of a burden aka how modern “learnable programming” is bs - I would suggest reading this when you’re having a particularly desperate day; it’s a morale boost more than anything but damn if it isn’t a morale boost when you’re just starting out and everything seems impossibly complex, not to mention adamant in its intentions to ruin your day. There are Bret Victor fans among Airtable’s very ranks, and this particular piece is probably his most iconic one to date.

There are probably better fetch API resources out there (yeah, yeah, you’re using remoteFetch, but same principles are giving you trouble here), but this one seems pretty decent and is relatively recent, so best I can do right now:

JavaScript Fetch Tutorial: Send HTTP Requests With React.JS and Async-Await Example | Techiediaries

What you need now is initiative to do your own research and come seeking help with theories and stories of hilariously catastrophic failures instead of just “oops, you forgot a semicolong there; also, need to write response.text() instead of response.text” kind of problems. People tend to ignore those, especially in low-code communities like this one, go figure. :slightly_smiling_face: