docs/guides/javascript-rendering

[giscus[bot]]

docs/guides/javascript-rendering

Your first steps into the world of scraping with Crawlee

https://crawlee.dev/docs/guides/javascript-rendering

[jonjibbed]

"Because View Page Source shows the original HTML, before any JavaScript executions."

There's the View Rendered Source extension for this exact problem. It shows raw source and rendered source side by side and a diff of the two. Very handy.

https://chrome.google.com/webstore/detail/view-rendered-source/ejgngohbdedoabanmclafpkoogegdpob

[jonboribbins]

JavaScript rendering is very fascinating and this was an interesting read. I appreciate the post.

[ashishagarwal2023]

I need help, how do I get the src of the img tag (rendered with javascript)? Here's the selector (copied from devtools of the element), theres no .src() or .getAttribute() function?
#root > div > div.login-section__card > form > div > div.form-section__captcha-field-container > div.form-section__captcha-box > div.form-section__captcha-image > img

[ashishagarwal2023]

Welp, VSCODE just shown in the autocomplete theres getAttribute in the locator, but here's what I recieve:
C:\Users\codew\crawl>node index.js
INFO PlaywrightCrawler: Starting the crawler.
ACTOR: null
INFO PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down.
INFO PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":4772,"requestsFinishedPerMinute":11,"requestsFailedPerMinute":0,"requestTotalDurationMillis":4772,"requestsTotal":1,"crawlerRuntimeMillis":5671}
INFO PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}

And my code:
import { PlaywrightCrawler } from "crawlee";

const crawler = new PlaywrightCrawler({
async requestHandler({ page }) {
// page.locator points to an element in the DOM
// using a CSS selector, but it does not access it yet.
const actorCard = page
.locator(
"#root > div > div.login-section__card > form > div > div.form-section__captcha-field-container > div.form-section__captcha-box > div.form-section__captcha-image > img"
)
.first();
// Upon calling one of the locator methods Playwright
// waits for the element to render and then accesses it.
const actorText = await actorCard.src;
console.log(SRC: ${actorText});
},
});

await crawler.run(["https://tathya.uidai.gov.in/access/login?role=resident"]);

[ashishagarwal2023]

Yeah but is there any way to go to 1 url -> click a button takes it to another URL with some cookies and session id -> then get src of img

(it will not hhave a src in the img if its no cookies)

[ashishagarwal2023]

This appears to be chatgpt frr "Certainly" Happy scraping, bruh