W3C WebDriver protocol support

[OndraM]

This is an issue to plan and track progress of implementing W3C WebDriver protocol into php-webdriver.

Currently, only legacy JsonWire (aka OSS) protocol is supported by php-webdriver. Selenium server 3.x translates the protocol if needed for the remote end (browser). However not everything is translated (like actions()), making some of the features not working with remote ends supporting only the W3C protocol (which is currently only GeckoDriver for Firefox 48+).

Since Selenium 3.5 the translation to old protocol is disabled by default. It still could be reenabled by starting the server like java -jar selenium-server-standalone-3.5.3.jar -enablePassThrough false, but this is only partial and temporary solution/workaround.

For Docker 🐳 users: use SE_OPTS environment variable to pass the enablePassThrough option to Selenium server. Ie. add -e "SE_OPTS=-enablePassThrough false" to docker run command.

The work would require eg.:

• add some more functional tests, to make sure we don't breake the old protocol while doing the necessary refactoring
• prepare handshake to detect whether the remote end is speaking JsonWire or W3C dialect (we need this to support the old browsers etc.)
• define common interface for Json and W3C protocol
• convert current implementation to use this interface
• implement adapter for the W3C protocol
• implement GeckodriverService or something like that to start geckodriver locally
  ...

Resources

• JsonWireProtocol vs. W3C WebDriver
• ProtocolHandshake in java bindings
• Handshake and capabilities resolving in python bindings

[OndraM]

I started working on https://github.com/facebook/php-webdriver/wiki/JsonWireProtocol-vs.-W3C-WebDriver . I'd like to summarize there the key protocol differences before we start actually implementing them in the code.

Any help with this overview document will be much appreciated!

[paulbriton]

Spec also contains changes about the JSON returned when finding an element.
The key ELEMENT is no longer used for referencing an element so we have an undefined index in RemoteWebDriver.php line 178. See this issue on selenium for more details.

Here is an example of the JSON returned by IEDriverServer (which now implements the spec in its latest version, 3.6):
{"state":"success","sessionId":null,"hCode":1164912850,"value":{"element-6066-11e4-a52e-4f735466cecf":"a1c4646c-8300-49f2-84b3-182911dc3eb6"},"class":"org.openqa.selenium.remote.Response","status":0}

[davidmintz]

It would be great if the docs were updated to warn people about this and potentially save them a couple hours of pain. Are you interested in a PR to this effect?

[OndraM]

@davidmintz I've recently added note about passThourghMode to readme. However we can do more, for sure :). What is on your mind?

I'm also hoping to have some time for the issue itself during November...

[davidmintz]

@OndraM sorry, I guess I missed that in the README, but I read your discussion about -enablePassThrough false, and tried it, but I am still exploding when I try to sendKeys(), with
Facebook\WebDriver\Exception\UnrecognizedExceptionException: Expected [object Undefined] undefined to be a string and I 'm unclear whether this rooted in the same issue as the undefined 'ELEMENT" index, or something else.

I do know it's making me crazy and I would love to help, just don't know if I have what it takes in terms of knowledge.

At the moment the only workaround that works for me is to use a very dated toolchain: Selenium 2.44 with Firefox 35. With the current versions of everything plus Chrome, I get "chrome is unreachable" and I've been all over the map trying to solve that one.

[alexmnv]

Note for Docker users: Selenium parameters are passed through SE_OPTS env variable. i.e, you need to add -e "SE_OPTS=-enablePassThrough false" to docker run command

[paulbriton]

Hi @OndraM ! Happy new year ;)

Any news on this issue ? I've found this Github repo which is based on an old version of this project and which implements the W3C protocol (at least it says so). Maybe it could help !

[pribbernow-marc]

For selenium-server-standalone version > 3.9.1: the option enablePassThrough doesn't work, just switch to version 3.8.1 to use this workaround for firefox

[OndraM]

Hi again, I'm so sorry for the delays 😞 and really unhappy with not pushing this forward recently, but my time resources are limited and I needed to spend lot of work on other job-stuff.

But I've prepared request for comments for php-webdriver 2.0 - see #657 , and I invite everyone interested to check it out and comment. So that once I'm in a short while back from upcoming vacation, we can finally start the hard work. Thanks everyone!

[OndraM]

📢 Current status update:

• The experimental W3C protocol support (W3C-compliant implementation, native geckodriver support #560) was finally merged to community branch! Thanks @dunglas and others involved! 💯 💯 💯
• If remote browser is capable of W3C protocol, it will now be automatically used instead of the old OSS JsonWireProtocol. This means eg. Chromedriver will now by default use W3C protocol. (This could still be disabled if you need to - see Enable W3C protocol for Chrome 🎉 #674 (comment) )
• This also means Geckodriver (ie. any new Firefox) could now be used! 🦊
• Even though library code coverage is far from being ideal, we have passing Travis test builds for various browsers and protocol combinations: Latest Chrome with JsonWire protocol, with W3C protocol, via Selenium server, directly via chromedriver; Firefox 45/47 using JsonWire protocol, latest Firefox 67 using W3C protocol directly via Geckodriver binary and also via Selenium server... So lets hope we won't break much things 🍀 .

What needs to be mentioned again - the W3C protocol support is experimental (and will remain experimental in upcoming php-webdriver 1.8). Some parts (like capabilities) are not yet implemented to fully conform W3C WebDriver protocol. However most of the stuff should work.

What will be the process now:

• Please test! You can easily update to the development version for the upcoming 1.8 release by changing dependency in your composer.json like this:

    "require": {
        "php-webdriver/webdriver": "^1.8@dev"
    }

and running composer update.
(Or you can just run composer require php-webdriver/webdriver:^1.8@dev).

• If you encounter any issues, please report them!
• This experimental W3C support will be released as part of php-webdriver 1.8, what should hopefully happen soon (if nothing bad happens).
• Further steps towards future php-webdriver 2.0 are discussed in php-webdriver 2.0 - RFC 💬 #657 . Also full support (not experimental) of W3C WebDriver will be part of php-webdriver 2.0.

🔥 🔥 🔥

[OndraM]

The community branch now contains further W3C protocol-related improvements:

• Capabilities are converted to W3C compatible ones
• Some window-realted commands (set size, maximize etc.) were added to those supported in W3C protocol mode

🔔 Everyone please test the community branch and report issues!
composer require php-webdriver/webdriver:^1.8@dev

[andrewnicols]

@OndraM, do you think that we may be able to release 1.8.0 shortly? I think we're nearly up to full compliance with the W3C spec (just a couple of missing commands which mostly have pull requests). Do you have a timeframe in mind?

[OndraM]

No exact timeframe, but I've created 1.8.0 milestone to track reamining issues. There is just a few of them, so it should not take long. And I don't think we necessarily need full W3C compliance, this would be aim of 2.0. We just need "good enough" implementation in 1.8 :-).

[OndraM]

📢 One more announcement for everyone watching this issue and waiting for W3C support.

As we move towards release 1.8.0, we successfully transferred the repository from Facebook (as explained in #730) and it is now available under new name. So in order to get any future updates of the package, you must update its name in require section of your composer.json file like this:

"require": {
-    "facebook/webdriver": "(version you use)",
+    "php-webdriver/webdriver": "(version you use)",
}

[OndraM]

🎉 Version 1.8.0 was released!

Thanks to all 14 community contributors who made this possible (in 120 commits)! 🥇 ❤️

🆕 Have look at complete changelog.

Version 1.8.0 includes experimental support of W3C WebDriver protocol, and it will be used by default if your browser supports it (recommended is latest Chrome/Firefox). If you encounter any issues with the W3C protocol, you can still explicitly disable the protocol and use the legacy "JsonWire OSS protocol", but please also report the issue here on GitHub.

Also don't forget to rename the library in your composer.json:

"require": {
-    "facebook/webdriver": "...",
+    "php-webdriver/webdriver": "^1.8.0",
}

before you run composer update.

If you use Selenium standalone server, make sure to use the latest version (either stable 3.141.59 or you can try alpha version of Selenium 4) and remove no longer needed enablePassThrough workaround you might used previously.

Happy testing!