JavaScript support or integration of Zombie

[func0der]

Hey there,

I loved the simplicity of your crawler. The only downside is, that it does not support javascript.

I am not completely sure, if adding JavaScript support makes it a Scraper, but it would be great.

Since Zombie already supports javascript it would be an idea to just use it as your gate to the outside world ;)

I would be glad to hear, what you are thinking about these suggestions.

Regards,
func0der

[ghost]

Since this http://googlewebmastercentral.blogspot.com.au/2014/05/understanding-web-pages-better.html

I think adding JavaScript support makes sense, as long as it's not enabled by default.
I will look into it, once i finished closing all issues and the release of CheerioJS build

[func0der]

That's great to hear :)

So 30 issues and one release ^^ I'll wait :)

[tmos]

I know it is from 2014, but any news about that ?
Thanks :)

[koolma]

I would also be interested in some news regarding JavaScript support.

[mike442144]

Aha, I'm glad that you guys keep watching this. However I'm too busy recently to add new features to crawler, and totally agree that if it is added it shouldn't be enabled by default.
We also need this feature in our projects, and I've tested Zombie about one year ago and found it hard to use and unstable. So does anybody use zombie recently? What do you think of it?
If we find Zombie unstable and unreliable we could also introduce other similar module.

[tmos]

Hey @mike442144

I haven't tested properly ZombieJS, but I guess the main concurent will be phantomJS. PhantomJS seems to be way more stable and tested than ZombieJS. I don't know what you think, but I think that the lightweightness of zombieJS is not an argument strong enough against a heavier but better field tester lib (phantomJS).

I don't really have the technical ability to work on the integration, but I would really love see this happen!

Cheers,
Tom

[mike442144]

Hi @tmos
I totally agree with you, because we have so many (100+) projects in our production environment. Once we have refactored the code we would start the work. But it's hard to tell the time table, maybe this quarter or this year. I'll come back here to update if any progress.

[tmos]

Apparently, some libs are using electron instead of phamtomjs. Example here : http://www.nightmarejs.org/
I have seen some other doing the same. Apparently it would be more modern, but you should keep in mind that electron don't seems to really sandbox code execution.

I don't really know, but if using electron instead of phantomJS seems to be a viable solution, and a way of using electron more modern stack and community.

[mike442144]

Cool, first time to see that somebody is using electron to do testing, and as you mentioned that security should be considered if using it. Better to have a look-up table to compare then.

[tmos]

Just found that Google have a nodejs headless googlechrome ready to use : https://github.com/GoogleChrome/puppeteer

Probably safer than electron!

[mike442144]

Fantastic! let's have a try.

[dzcpy]

Hi, how's the progress on this one?

[mike442144]

@andyhu Hey guy, we're now working on refactor the tests first ,and refactor the core codes after. If possible I think we'd like to introduce this feature as a middleware or plugin which avoids to slow down the installing and running time. Pls see the current progress on branch 20180611_updatetestwithnock, and do not hesitate to fork if you're also interested in it.

[poorren]

Hi, how about the progress of js support now?