Twitter’s web app was refactored and I’ve re-jigged my web clipper as best I can. This web clipper will take a Twitter user’s profile information into a table with fields for:
Name
Profile from page URL - to be trimmed
Avatar
Bio from selected text
The refactored mark-up in Twitter web app means this clip action needs a little extra work.
The steps I follow:
- go to someone’s Twitter profile eg https://twitter.com/khoi
- select the text on their bio
- click their avatar to enlarge it (this exposes a CSS selector for clipping the image, and appends /photo to the URL eg https://twitter.com/khoi/photo)
- activate Airtable web clipper action
- trim off the /photo from the page URL
- tidy up any undesirable new lines in the Bio text produced by @mentions or #tags
Here’s the clip action recipe for Twitter profiles:
{
"schemaVersion": 3,
"fieldMappings": [
{
"fieldName": "Name",
"fieldType": "singleLineText",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span"
}
}
},
{
"fieldName": "Profile",
"fieldType": "url",
"defaultValue": {
"type": "pageUrl",
"opts": null
}
},
{
"fieldName": "Avatar",
"fieldType": "multipleAttachments",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "img[draggable=\"true\"]"
}
}
},
{
"fieldName": "Bio",
"fieldType": "multilineText",
"defaultValue": {
"type": "selectedText",
"opts": null
}
}
]
}
I’ve extended this clipper to also scrape from LinkedIn profile pages. Each CSS selector now has a comma separator (Twitter, LinkedIn).
Here’s an annotated screenshot outlining where the CSS selectors draw from.
This example lacks a profile background image. The selector avoids targeting an image with alt="Background Image" anyway.
Lastly, I’m using the About paragraph to populate my Bio field. If you prefer the one-liner that appears after their name, switch the selector from main p[class*="about__summary"] to main section h2
Clip action:
{
"schemaVersion": 3,
"fieldMappings": [
{
"fieldName": "Name",
"fieldType": "singleLineText",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "h2[aria-level=\"2\"][role=\"heading\"] div div div span, main section li"
}
}
},
{
"fieldName": "Profile",
"fieldType": "url",
"defaultValue": {
"type": "pageUrl",
"opts": null
}
},
{
"fieldName": "Avatar",
"fieldType": "multipleAttachments",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div.css-1dbjc4n.r-obd0qt.r-18u37iz.r-1w6e6rj.r-1wtj0ep > a > div.css-1dbjc4n.r-1adg3ll.r-1udh08x > div.r-1p0dtai.r-1pi2tsx.r-1d2f490.r-u8s1d.r-ipm5af.r-13qz1uu > div > img, main section img:not([alt=\"Background Image\"])"
}
}
},
{
"fieldName": "Bio",
"fieldType": "multilineText",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "[data-testid=\"UserDescription\"], main p[class*=\"about__summary\"]"
}
}
},
{
"fieldName": "Location",
"fieldType": "singleLineText",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "div.css-1dbjc4n.r-14lw9ot.r-1tlfku8.r-1ljd8xs.r-13l2t4g.r-1phboty.r-1jgb5lz.r-11wrixw.r-61z16t.r-1ye8kvj.r-13qz1uu.r-184en5c > div > div:nth-child(2) > div > div > div:nth-child(1) > div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > span:nth-child(1) > span, main section ul ~ ul li"
}
}
},
{
"fieldName": "Website",
"fieldType": "url",
"defaultValue": {
"type": "cssSelector",
"opts": {
"cssSelector": "div.css-1dbjc4n.r-ku1wi2.r-1j3t67a.r-m611by > div:nth-child(4) > div > a"
}
}
}
]
}
