You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
teammakdi opened this issue
Apr 8, 2024
· 1 comment
Labels
featureIssues that represent new features or improvements to existing features.t-c&cTeam covering store and finance matters.t-consoleIssues with this label are in the ownership of the console team.t-toolingIssues with this label are in the ownership of the tooling team.
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/http (HttpCrawler)
Feature
Hi, I see with both HttpCrawler and PuppeteerCrawler, ProxyConfiguration is supported which needs a HTTP proxy server. However my use case is to use the secondary IP address for crawling purposes.
Motivation
Raw axios supports requesting from a secondary IP address present on the machine. Example
consthttpsAgent=newhttps.Agent({localAddress: 'x.x.x.x',localPort: xxxx});awaitaxios.get('https://api.ipify.org',{
httpsAgent
}).then(response=>{console.log('HTTPS Agent: ',response.data);// prints secondary IP address}).catch(err=>{console.error(err);})
Was wondering if it could be possible with the crawlee HttpCrawler i.e. with got library. Not sure if it would be feasible with the PuppeteerCrawler.
Ideal solution or implementation, and any additional constraints
Alternative solutions or implementations
No response
Other context
No response
The text was updated successfully, but these errors were encountered:
teammakdi
added
the
feature
Issues that represent new features or improvements to existing features.
label
Apr 8, 2024
mtrunkat
added
t-tooling
Issues with this label are in the ownership of the tooling team.
t-console
Issues with this label are in the ownership of the console team.
t-c&c
Team covering store and finance matters.
labels
Apr 10, 2024
featureIssues that represent new features or improvements to existing features.t-c&cTeam covering store and finance matters.t-consoleIssues with this label are in the ownership of the console team.t-toolingIssues with this label are in the ownership of the tooling team.
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/http (HttpCrawler)
Feature
Hi, I see with both HttpCrawler and PuppeteerCrawler, ProxyConfiguration is supported which needs a HTTP proxy server. However my use case is to use the secondary IP address for crawling purposes.
Motivation
Raw axios supports requesting from a secondary IP address present on the machine. Example
Was wondering if it could be possible with the crawlee HttpCrawler i.e. with got library. Not sure if it would be feasible with the PuppeteerCrawler.
Ideal solution or implementation, and any additional constraints
Alternative solutions or implementations
No response
Other context
No response
The text was updated successfully, but these errors were encountered: