Twitter profile clipper

[Vernon_Fowler]

Twitter’s web app was refactored and I’ve re-jigged my web clipper as best I can. This web clipper will take a Twitter user’s profile information into a table with fields for:

Name
Profile from page URL - to be trimmed
Avatar
Bio from selected text

The refactored mark-up in Twitter web app means this clip action needs a little extra work.
The steps I follow:

1. go to someone’s Twitter profile eg https://twitter.com/khoi
2. select the text on their bio
3. click their avatar to enlarge it (this exposes a CSS selector for clipping the image, and appends /photo to the URL eg https://twitter.com/khoi/photo)
4. activate Airtable web clipper action
5. trim off the /photo from the page URL
6. tidy up any undesirable new lines in the Bio text produced by @mentions or #tags

Here’s the clip action recipe for Twitter profiles:

{
    "schemaVersion": 3,
    "fieldMappings": [
        {
            "fieldName": "Name",
            "fieldType": "singleLineText",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span"
                }
            }
        },
        {
            "fieldName": "Profile",
            "fieldType": "url",
            "defaultValue": {
                "type": "pageUrl",
                "opts": null
            }
        },
        {
            "fieldName": "Avatar",
            "fieldType": "multipleAttachments",
            "defaultValue": {
                "type": "cssSelector",
                "opts": {
                    "cssSelector": "img[draggable=\"true\"]"
                }
            }
        },
        {
            "fieldName": "Bio",
            "fieldType": "multilineText",
            "defaultValue": {
                "type": "selectedText",
                "opts": null
            }
        }
    ]
}

[Vernon_Fowler]

That’s correct @Anne_Marie_Helwaser That latest clip action works on LinkedIn. :slightly_smiling_face:

When they eventually change their HTML/CSS I’m sure someone will contribute an updated clip action right here in this thread. :winking_face: Thanks to all who have contributed / will contribute improvements and iterations. :thumbs_up:

[Boyan_One]

That’s a great post and the code works great.

Is it possible to use the web clipper to select text (profile) from Linkedin search results and copy the data like (first/last name, title, picture URL and profile URL) to the Airtable?

I’d like to do this without visiting the profile.

[Vernon_Fowler]

Hi @Boyan_One,

It may be possible to extract from search results but I’m struggling to figure out how. My assumption is you want to grab a different search result each time you perform a clip (6th result this time, 4th another time, and then the 9th on another occasion).

I think extracting fields automatically would depend on the CSS selectors for a specific search result index (eg the first result) to be effective. I don’t know how it might work otherwise.

Anyone else have an idea how this might work?

[Adam_W]

Hi @Vernon_Fowler

Thank you for putting this together. It’s been incredibly helpful. There’s only one thing I’m struggling with which I hope you can help with.
Part of the data I need to grab from twitter is the follow count.

This lives here:
span class=“css-901oao css-16my406 r-1qd0xha r-ad9z0x r-bcqeeo r-qvutc0”

However, no matter what I do, I can’t seem to get the web clipper to recognize this.

What am I doing wrong?

[Vernon_Fowler]

Hi @Adam_W and welcome to the community.

This was tricky. There are 2 spans with those classes: the following and the followers count links.

Be sure your Airtable fields are Single line text as Brad Frost’s Twitter account has 113.9K followers. If Airtable fields are number, you’ll lose the K for thousand!

The trick is to include the unique a tag attribute selector before your span, eg for followers:

[href*="followers"] span.css-901oao.css-16my406.r-18jsvk2.r-1qd0xha.r-b88u0q.r-ad9z0x.r-bcqeeo.r-qvutc0

or if you want the following count

[href*="following"] span.css-901oao.css-16my406.r-18jsvk2.r-1qd0xha.r-b88u0q.r-ad9z0x.r-bcqeeo.r-qvutc0

If you need a pure number rather than text, try some functions in a formula field based on these. Alternatively figure out how to get the hover text which appears more numeric (113,910 in the case of Brad Frost).

[jamesvclements]

Thanks all for the contributions! It looks like the clipper is no longer working for Twitter profile images Here’s an updated selector to grab that image (no need to click image first):

[data-testid="primaryColumn"]  .r-m611by a[href$="photo"] img

[Vernon_Fowler]

@jamesvclements thanks for the heads up on the clipper no longer working for Twitter profile images. :raised_hands:t2:

A slightly more robust selector that does not rely on classes is:

img[src*="200x200"]

This clips the 200x200 size image from a Twitter profile. And if you want to clip LinkedIn profile images to the same table, have both selectors:

img[src*="200x200"], main section img:not([alt="Background Image"])

Want a larger image (from either Twitter or LinkedIn)?

1. Start your clip action.
2. Remove the existing image from the clipper.
3. On the web page, click the profile avatar link to show the larger image.
4. Back on the clipper, choose Add image from page and select the larger image. :v:t2:

[Edmund_Amoye]

Hey there folks. I just want to use an automation in airtable to pull the proilfe image from a URL and return it as an image. Has anyone done this?