Help

Re: Twitter profile clipper

3049 0
cancel
Showing results for 
Search instead for 
Did you mean: 
Vernon_Fowler
7 - App Architect
7 - App Architect

Twitter’s web app was refactored and I’ve re-jigged my web clipper as best I can. This web clipper will take a Twitter user’s profile information into a table with fields for:

Name
Profile from page URL - to be trimmed
Avatar
Bio from selected text

The refactored mark-up in Twitter web app means this clip action needs a little extra work.
The steps I follow:

  1. go to someone’s Twitter profile eg https://twitter.com/khoi
  2. select the text on their bio
  3. click their avatar to enlarge it (this exposes a CSS selector for clipping the image, and appends /photo to the URL eg https://twitter.com/khoi/photo)
  4. activate Airtable web clipper action
  5. trim off the /photo from the page URL
  6. tidy up any undesirable new lines in the Bio text produced by @mentions or #tags

Twitter profile Airtable web clip action.png
Here’s the clip action recipe for Twitter profiles:

{
    "schemaVersion": 3,
    "fieldMappings": [
        {
            "fieldName": "Name",
            "fieldType": "singleLineText",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span"
                }
            }
        },
        {
            "fieldName": "Profile",
            "fieldType": "url",
            "defaultValue": {
                "type": "pageUrl",
                "opts": null
            }
        },
        {
            "fieldName": "Avatar",
            "fieldType": "multipleAttachments",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "img[draggable=\"true\"]"
                }
            }
        },
        {
            "fieldName": "Bio",
            "fieldType": "multilineText",
            "defaultValue": {
                "type": "selectedText",
                "opts": null
            }
        }
    ]
}
17 Replies 17
Andrew_Wingrave
6 - Interface Innovator
6 - Interface Innovator

Hey Vernon, I updated this slightly to auto-copy the description from the page. Tried a few ways but couldn’t pull the avatar without opening up the photo page - A bit of a pain, but this worked for my purposes. Thanks

{
"schemaVersion": 3,
"fieldMappings": [
    {
        "fieldName": "Brand Name",
        "fieldType": "singleLineText",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span"
            }
        }
    },
    {
        "fieldName": "Twitter Handle",
        "fieldType": "multilineText",
        "defaultValue": {
            "type": "pageUrl",
            "opts": null
        }
    },
    {
        "fieldName": "Image Logo",
        "fieldType": "multipleAttachments",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "img[draggable=\"true\"]"
            }
        }
    },
    {
        "fieldName": "Bio",
        "fieldType": "singleLineText",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "[data-testid=\"UserDescription\"]"
            }
        }
    }
]

}

chrismessina
6 - Interface Innovator
6 - Interface Innovator

I was able to make some improvements by digging in deep, which might make my solutions brittle over time. Still, you can now pull the Profile Photo, Location, and URL just from a typical profile page:

{
"schemaVersion": 3,
"fieldMappings": [
    {
        "fieldName": "Name",
        "fieldType": "singleLineText",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span"
            }
        }
    },
    {
        "fieldName": "Twitter",
        "fieldType": "url",
        "defaultValue": {
            "type": "pageUrl",
            "opts": null
        }
    },
    {
        "fieldName": "Profile Photo",
        "fieldType": "multipleAttachments",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div.css-1dbjc4n.r-obd0qt.r-18u37iz.r-1w6e6rj.r-1wtj0ep > a > div.css-1dbjc4n.r-1adg3ll.r-1udh08x > div.r-1p0dtai.r-1pi2tsx.r-1d2f490.r-u8s1d.r-ipm5af.r-13qz1uu > div > img"
            }
        }
    },
    {
        "fieldName": "Bio",
        "fieldType": "multilineText",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "[data-testid=\"UserDescription\"]"
            }
        }
    },
    {
        "fieldName": "Location",
        "fieldType": "singleLineText",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > span:nth-child(1) > span"
            }
        }
    },
    {
        "fieldName": "Website",
        "fieldType": "url",
        "defaultValue": {
            "type": "cssSelector",
            "opts": {
                "cssSelector": "div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > a"
            }
        }
    }
]

}

Thank you @Andrew_Wingrave for adding the cssSelector for the Bio field from [data-testid="UserDescription"]. I’m learning more about data-* custom data attributes. :winking_face:

Hi @chrismessina and welcome to our community :wave:

Wow, what an upgrade to the Twitter profile clipper. Nice digging! :clap:
I like the cssSelector that pulls in the avatar without needing the extra clicking workaround. Nice. :winking_face:
What a bonus to grab the location and Website fields too!

I like these upgrades so much I retro-fitted some of records in my Airtable (clipping with the upgraded web clipper, then using a Dedupe block for matches on the profile URL).

Vernon_Fowler
7 - App Architect
7 - App Architect

I’ve extended this clipper to also scrape from LinkedIn profile pages. Each CSS selector now has a comma separator (Twitter, LinkedIn).

Here’s an annotated screenshot outlining where the CSS selectors draw from.
Khoi Vinh   LinkedIn
This example lacks a profile background image. The selector avoids targeting an image with alt="Background Image" anyway.

Lastly, I’m using the About paragraph to populate my Bio field. If you prefer the one-liner that appears after their name, switch the selector from main p[class*="about__summary"] to main section h2

Clip action:

{
    "schemaVersion": 3,
    "fieldMappings": [
        {
            "fieldName": "Name",
            "fieldType": "singleLineText",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span, main section li"
                }
            }
        },
        {
            "fieldName": "Profile",
            "fieldType": "url",
            "defaultValue": {
                "type": "pageUrl",
                "opts": null
            }
        },
        {
            "fieldName": "Avatar",
            "fieldType": "multipleAttachments",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div.css-1dbjc4n.r-obd0qt.r-18u37iz.r-1w6e6rj.r-1wtj0ep > a > div.css-1dbjc4n.r-1adg3ll.r-1udh08x > div.r-1p0dtai.r-1pi2tsx.r-1d2f490.r-u8s1d.r-ipm5af.r-13qz1uu > div > img, main section img:not([alt=\"Background Image\"])"
                }
            }
        },
        {
            "fieldName": "Bio",
            "fieldType": "multilineText",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "[data-testid=\"UserDescription\"], main p[class*=\"about__summary\"]"
                }
            }
        },
        {
            "fieldName": "Location",
            "fieldType": "singleLineText",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > span:nth-child(1) > span, main section ul ~ ul li"
                }
            }
        },
        {
            "fieldName": "Website",
            "fieldType": "url",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > a"
                }
            }
        }
    ]
}
Anne_Marie_Helw
4 - Data Explorer
4 - Data Explorer

I am a beginner and my question may seem stupid: using the web clipper, I “import clipaction” first, correct?
and then I had pasted your script inside “snippet clipper”
it returns Incorrect.
Could you tell me why?

Hi @Anne_Marie_Helwaser! Here are the steps to import any of the clip actions above:

  1. Open your web clipper block
  2. Click the “Import” button, which is underneath the list of fields in your web clipper block
  3. Paste the snippet above into the text area, then click "Import"
  4. At this next step, you’ll map fields from the snippet to fields in your own table! As long as you setup at least one field, you’ll be able to continue with the import. For example, in the snippet Vernon shared for clipping Twitter profiles, you would map the “Name” field to whichever column in your table you’d like to save names to. If your table has a field to map to everything defined in the snippet, you can map them all!
  5. Click “Save”, then get clipping!

There are some logical restrictions with field mapping - for example, you can only map a field that fetches images to an Attachment field. For the most part, though, you can define mappings as you wish!

Let me know if you’re still running into issues with the clip action importing!

Cheers.

Anne-Marie_Helw
4 - Data Explorer
4 - Data Explorer

Thank you so much!
One more question… just to make sure (this was a while ago): I can clip from LinkedIn profile with the snippet or was it for Twitter only( see above Vernon Fowler in the conversation mentioning LinkedIn which is better for me

Hi @Anne-Marie_Helwaser

Great news: The latest iteration of the clip action works for both LinkedIn and Twitter. :winking_face:

The clip action from late December includes comma separated CSS selectors that grab relevant text strings from either site. For example, the fieldName “Bio” has:

"cssSelector": "[data-testid=\"UserDescription\"], main p[class*=\"about__summary\"]"
The comma between selector elements acts like a match all.
When on a Twitter profile, [data-testid=\"UserDescription\"] will find a match.
When on a LinkedIn profile, main p[class*=\"about__summary\"] will find a match.

When either Twitter or LinkedIn change their sites to use different HTML, these selectors may need refactoring.


Using commas in CSS selectors a clip action can be extended to support multiple sites. This capability of using commas to specify multiple selectors doesn’t appear in the Airtable documentation: Creating CSS selectors for the web clipper block. However there is a link at the end pointing to CSS Selectors Reference which includes CSS element,element Selector

Anne_Marie_Helw
4 - Data Explorer
4 - Data Explorer

Great ! Thanks,
meaning for now I don’t have to change anything as long as I use it on LinkedIn, and they don’t change their sites to use different HTML correct?

That’s correct @Anne_Marie_Helwaser That latest clip action works on LinkedIn. :slightly_smiling_face:

When they eventually change their HTML/CSS I’m sure someone will contribute an updated clip action right here in this thread. :winking_face: Thanks to all who have contributed / will contribute improvements and iterations. :thumbs_up:

Boyan_One
4 - Data Explorer
4 - Data Explorer

That’s a great post and the code works great.

Is it possible to use the web clipper to select text (profile) from Linkedin search results and copy the data like (first/last name, title, picture URL and profile URL) to the Airtable?

I’d like to do this without visiting the profile.

Hi @Boyan_One,

It may be possible to extract from search results but I’m struggling to figure out how. My assumption is you want to grab a different search result each time you perform a clip (6th result this time, 4th another time, and then the 9th on another occasion).

I think extracting fields automatically would depend on the CSS selectors for a specific search result index (eg the first result) to be effective. I don’t know how it might work otherwise.

Anyone else have an idea how this might work?

Adam_W
4 - Data Explorer
4 - Data Explorer

Hi @Vernon_Fowler

Thank you for putting this together. It’s been incredibly helpful. There’s only one thing I’m struggling with which I hope you can help with.
Part of the data I need to grab from twitter is the follow count.

This lives here:
span class=“css-901oao css-16my406 r-1qd0xha r-ad9z0x r-bcqeeo r-qvutc0”

However, no matter what I do, I can’t seem to get the web clipper to recognize this.

What am I doing wrong?

Hi @Adam_W and welcome to the community.

This was tricky. There are 2 spans with those classes: the following and the followers count links.

Be sure your Airtable fields are Single line text as Brad Frost’s Twitter account has 113.9K followers. If Airtable fields are number, you’ll lose the K for thousand!

The trick is to include the unique a tag attribute selector before your span, eg for followers:

[href*="followers"] span.css-901oao.css-16my406.r-18jsvk2.r-1qd0xha.r-b88u0q.r-ad9z0x.r-bcqeeo.r-qvutc0

followers

or if you want the following count

[href*="following"] span.css-901oao.css-16my406.r-18jsvk2.r-1qd0xha.r-b88u0q.r-ad9z0x.r-bcqeeo.r-qvutc0

If you need a pure number rather than text, try some functions in a formula field based on these. Alternatively figure out how to get the hover text which appears more numeric (113,910 in the case of Brad Frost).

Thanks all for the contributions! It looks like the clipper is no longer working for Twitter profile images Here’s an updated selector to grab that image (no need to click image first):

[data-testid="primaryColumn"]  .r-m611by a[href$="photo"] img

@jamesvclements thanks for the heads up on the clipper no longer working for Twitter profile images. :raised_hands:t2:

A slightly more robust selector that does not rely on classes is:

img[src*="200x200"]

This clips the 200x200 size image from a Twitter profile. And if you want to clip LinkedIn profile images to the same table, have both selectors:

img[src*="200x200"], main section img:not([alt="Background Image"])

Want a larger image (from either Twitter or LinkedIn)?

  1. Start your clip action.
  2. Remove the existing image from the clipper.
  3. On the web page, click the profile avatar link to show the larger image.
  4. Back on the clipper, choose Add image from page and select the larger image. :v:t2:

2021-01-08_14h08_09

Edmund_Amoye
4 - Data Explorer
4 - Data Explorer

Hey there folks. I just want to use an automation in airtable to pull the proilfe image from a URL and return it as an image. Has anyone done this?