NotImplementedError on Windows Python 3.8+

[anon88391]

Running proxybroker from CLI has been broken for me on any Python version >= 3.8

Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\pycares_init_.py", line 99, in sock_state_cb
sock_state_cb(socket_fd, readable, writable)
File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\aiodns_init.py", line 111, in _sock_state_cb
self.loop.add_reader(fd, self._handle_event, fd, READ)
File "c:\users\user\appdata\local\programs\python\python38\lib\asyncio\events.py", line 501, in add_reader
raise NotImplementedError

Apparently, Python 3.8 introduced a backwards-incompatible change to asyncio.
https://docs.python.org/3/library/asyncio-platforms.html#asyncio-platform-support
https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.add_reader

Some people suggested adding the following code to fix it:

import sys

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Checklist

• proxybroker/cli.py

• At the top of the file, after the import statements, add the following code:

import sys
import asyncio

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

This code checks if the current platform is Windows ('win32'), and if so, it sets the event loop policy to WindowsSelectorEventLoopPolicy.

[anon88391]

Didn't mean to close the issue.

Related:
saghul/aiodns#86

[BukuBukuChagma]

running into same problem. Any solution yet?

[ziloka]

I found a solution to this. The issue is relating to aiodns not really being maintained anymore. The last time the source code was updated was 2 years ago. I believe the current solution to this is to remove aiodns and replace it with dnspython A library which is kept more up to date.

Hopefully I will be able to make a PR fixing this issue

[sweep-ai]

Here's the PR! #138.

⚡ Sweep Free Trial: I used GPT-4 to create this ticket. You have 4 GPT-4 tickets left for the month and 1 for the day. For more GPT-4 tickets, visit our payment portal. To retrigger Sweep edit the issue.

• Install Sweep Configs: Pull Request

---

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

proxybroker2/proxybroker/__init__.py

Lines 1 to 45 in cadfc4c

	"""
	Copyright © 2015-2018 Constverum <constverum@gmail.com>. All rights reserved.
	Licensed under the Apache License, Version 2.0 (the "License");
	you may not use this file except in compliance with the License.
	You may obtain a copy of the License at
	http://www.apache.org/licenses/LICENSE-2.0
	Unless required by applicable law or agreed to in writing, software
	distributed under the License is distributed on an "AS IS" BASIS,
	WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	See the License for the specific language governing permissions and
	limitations under the License.
	"""
	__title__ = 'ProxyBroker'
	__package__ = 'proxybroker'
	__version__ = '0.4.0'
	__short_description__ = '[Finder/Checker/Server] Finds public proxies from multiple sources and concurrently checks them. Supports HTTP(S) and SOCKS4/5.' # noqa
	__author__ = 'Constverum'
	__author_email__ = 'constverum@gmail.com'
	__url__ = 'https://github.com/constverum/ProxyBroker'
	__license__ = 'Apache License, Version 2.0'
	__copyright__ = 'Copyright 2015-2018 Constverum'
	import logging # noqa
	import warnings # noqa
	from .api import Broker # noqa
	from .checker import Checker # noqa
	from .judge import Judge # noqa
	from .providers import Provider # noqa
	from .proxy import Proxy # noqa
	from .server import ProxyPool, Server # noqa
	logger = logging.getLogger('asyncio')
	logger.addFilter(logging.Filter('has no effect when using ssl'))
	warnings.simplefilter('always', UserWarning)
	warnings.simplefilter('once', DeprecationWarning)
	__all__ = (Proxy, Judge, Provider, Checker, Server, ProxyPool, Broker)

https://github.com/bluet/proxybroker2/blob/cadfc4c589b55de481e506a14d8c7f6ab9b72cdd/proxybroker/data/__init__.py#L1-L0

https://github.com/bluet/proxybroker2/blob/cadfc4c589b55de481e506a14d8c7f6ab9b72cdd/tests/__init__.py#L1-L0

proxybroker2/proxybroker/negotiators.py

Lines 1 to 90 in cadfc4c

	import asyncio
	import struct
	from abc import ABC, abstractmethod
	from socket import inet_aton
	from .errors import BadResponseError, BadStatusError
	from .utils import get_headers, get_status_code
	__all__ = [
	'Socks5Ngtr',
	'Socks4Ngtr',
	'Connect80Ngtr',
	'Connect25Ngtr',
	'HttpsNgtr',
	'HttpNgtr',
	'NGTRS',
	]
	SMTP_READY = 220
	def _CONNECT_request(host, port, **kwargs):
	kwargs.setdefault('User-Agent', get_headers()['User-Agent'])
	kw = {
	'host': host,
	'port': port,
	'headers': '\r\n'.join(('%s: %s' % (k, v) for k, v in kwargs.items())),
	}
	req = (
	(
	'CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n'
	'{headers}\r\nConnection: keep-alive\r\n\r\n'
	)
	.format(**kw)
	.encode()
	)
	return req
	class BaseNegotiator(ABC):
	"""Base Negotiator."""
	name = None
	check_anon_lvl = False
	use_full_path = False
	def __init__(self, proxy):
	self._proxy = proxy
	@abstractmethod
	async def negotiate(self, **kwargs):
	"""Negotiate with proxy."""
	class Socks5Ngtr(BaseNegotiator):
	"""SOCKS5 Negotiator."""
	name = 'SOCKS5'
	async def negotiate(self, **kwargs):
	await self._proxy.send(struct.pack('3B', 5, 1, 0))
	resp = await self._proxy.recv(2)
	if not isinstance(resp, (bytes, str)):
	raise TypeError(f"{type(resp).__name__} is not supported")
	if resp[0] == 0x05 and resp[1] == 0xFF:
	self._proxy.log('Failed (auth is required)', err=BadResponseError)
	raise BadResponseError
	elif resp[0] != 0x05 or resp[1] != 0x00:
	self._proxy.log('Failed (invalid data)', err=BadResponseError)
	raise BadResponseError
	bip = inet_aton(kwargs.get('ip'))
	port = kwargs.get('port', 80)
	await self._proxy.send(struct.pack('>8BH', 5, 1, 0, 1, *bip, port))
	resp = await self._proxy.recv(10)
	if resp[0] != 0x05 or resp[1] != 0x00:
	self._proxy.log('Failed (invalid data)', err=BadResponseError)
	raise BadResponseError
	else:
	self._proxy.log('Request is granted')
	class Socks4Ngtr(BaseNegotiator):
	"""SOCKS4 Negotiator."""
	name = 'SOCKS4'

proxybroker2/proxybroker/api.py

Lines 31 to 147 in cadfc4c

	:param int timeout: (optional) Timeout of a request in seconds
	:param int max_conn:
	(optional) The maximum number of concurrent checks of proxies
	:param int max_tries:
	(optional) The maximum number of attempts to check a proxy
	:param list judges:
	(optional) Urls of pages that show HTTP headers and IP address.
	Or :class:`~proxybroker.judge.Judge` objects
	:param list providers:
	(optional) Urls of pages where to find proxies.
	Or :class:`~proxybroker.providers.Provider` objects
	:param bool verify_ssl:
	(optional) Flag indicating whether to check the SSL certificates.
	Set to True to check ssl certifications
	:param loop: (optional) asyncio compatible event loop
	:param stop_broker_on_sigint: (optional) whether set SIGINT signal on broker object.
	Useful for a thread other than main thread.
	.. deprecated:: 0.2.0
	Use :attr:`max_conn` and :attr:`max_tries` instead of
	:attr:`max_concurrent_conn` and :attr:`attempts_conn`.
	"""
	def __init__(
	self,
	queue=None,
	timeout=8,
	max_conn=200,
	max_tries=3,
	judges=None,
	providers=None,
	verify_ssl=False,
	loop=None,
	stop_broker_on_sigint=True,
	**kwargs,
	):
	self._loop = loop or asyncio.get_event_loop_policy().get_event_loop()
	self._proxies = queue or asyncio.Queue()
	self._resolver = Resolver(loop=self._loop)
	self._timeout = timeout
	self._verify_ssl = verify_ssl
	self.unique_proxies = {}
	self._all_tasks = []
	self._checker = None
	self._server = None
	self._limit = 0 # not limited
	self._countries = None
	max_concurrent_conn = kwargs.get('max_concurrent_conn')
	if max_concurrent_conn:
	warnings.warn(
	'`max_concurrent_conn` is deprecated, use `max_conn` instead',
	DeprecationWarning,
	)
	if isinstance(max_concurrent_conn, asyncio.Semaphore):
	max_conn = max_concurrent_conn._value
	else:
	max_conn = max_concurrent_conn
	attempts_conn = kwargs.get('attempts_conn')
	if attempts_conn:
	warnings.warn(
	'`attempts_conn` is deprecated, use `max_tries` instead',
	DeprecationWarning,
	)
	max_tries = attempts_conn
	# The maximum number of concurrent checking proxies
	self._on_check = asyncio.Queue(maxsize=max_conn)
	self._max_tries = max_tries
	self._judges = judges
	self._providers = [
	p if isinstance(p, Provider) else Provider(p)
	for p in (providers or PROVIDERS)
	]
	if stop_broker_on_sigint:
	try:
	self._loop.add_signal_handler(signal.SIGINT, self.stop)
	# add_signal_handler() is not implemented on Win
	# https://docs.python.org/3.5/library/asyncio-eventloops.html#windows
	except NotImplementedError:
	pass
	async def grab(self, *, countries=None, limit=0):
	"""Gather proxies from the providers without checking.
	:param list countries: (optional) List of ISO country codes
	where should be located proxies
	:param int limit: (optional) The maximum number of proxies
	:ref:`Example of usage <proxybroker-examples-grab>`.
	"""
	self._countries = countries
	self._limit = limit
	task = asyncio.ensure_future(self._grab(check=False))
	self._all_tasks.append(task)
	async def find(
	self,
	*,
	types=None,
	data=None,
	countries=None,
	post=False,
	strict=False,
	dnsbl=None,
	limit=0,
	**kwargs,
	):
	"""Gather and check proxies from providers or from a passed data.
	:ref:`Example of usage <proxybroker-examples-find>`.
	:param list types:
	Types (protocols) that need to be check on support by proxy.
	Supported: HTTP, HTTPS, SOCKS4, SOCKS5, CONNECT:80, CONNECT:25

I also found the following external resources that might be helpful:

Summaries of links found in the content:

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.add_reader:

The page is the documentation for the Event Loop in Python's asyncio library. It provides information on how to work with the event loop, including methods for running and stopping the loop, scheduling callbacks, creating futures and tasks, opening network connections, creating network servers, transferring files, and more.

The page also includes examples of how to use the event loop, such as scheduling callbacks with call_soon() and call_later(), watching file descriptors for read events, setting signal handlers for SIGINT and SIGTERM, and running subprocesses.

In relation to the problem mentioned by the user, the page does not provide a direct solution. However, it does mention that asyncio ships with two different event loop implementations: SelectorEventLoop and ProactorEventLoop. By default, asyncio is configured to use SelectorEventLoop on Unix and ProactorEventLoop on Windows. The user may need to check if their code is compatible with the event loop implementation they are using.

Additionally, the page provides information on how to set the event loop policy using asyncio.set_event_loop_policy() to specify a custom event loop implementation. This may be relevant to the user's problem if they need to switch to a different event loop implementation to resolve the compatibility issue.

The code snippet suggested by some people to fix the issue involves setting the event loop policy to asyncio.WindowsSelectorEventLoopPolicy() if the platform is Windows. This may be a workaround for the compatibility issue mentioned by the user.

Overall, the page provides a comprehensive overview of the event loop in asyncio and includes examples that may be helpful in understanding and solving the problem.

https://docs.python.org/3/library/asyncio-platforms.html#asyncio-platform-support:

The page titled "Platform Support" in the Python 3.11.4 documentation discusses the platform-specific differences and limitations of the asyncio module. It mentions that the asyncio module is designed to be portable, but some platforms have subtle differences and limitations due to their underlying architecture and capabilities.

The page provides information on platform support for all platforms, Windows, and macOS. It states that on all platforms, the methods loop.add_reader() and loop.add_writer() cannot be used to monitor file I/O.

For Windows, it mentions that the ProactorEventLoop is now the default event loop starting from version 3.8. It also lists the methods and features that are not supported on Windows, such as loop.create_unix_connection(), loop.create_unix_server(), loop.add_signal_handler(), and loop.remove_signal_handler(). It further explains the limitations of the SelectorEventLoop and ProactorEventLoop on Windows.

Regarding macOS, it states that modern macOS versions are fully supported. However, on macOS versions 10.6, 10.7, and 10.8, the default event loop uses selectors.KqueueSelector, which does not support character devices on these versions. It provides an example of manually configuring the SelectorEventLoop to use SelectSelector or PollSelector to support character devices on these older versions of macOS.

In summary, the page provides information on the platform-specific differences and limitations of the asyncio module, including details about support on all platforms, Windows, and macOS. It does not specifically address the issue mentioned by the user, but it may provide some insights into the problem. Additionally, the page does not include the suggested code snippet to fix the issue.

---

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path	Proposed Changes
proxybroker/cli.py	Modify proxybroker/cli.py with contents: • At the top of the file, after the import statements, add the following code: ```python import sys import asyncio if sys.platform == 'win32': asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) ``` This code checks if the current platform is Windows ('win32'), and if so, it sets the event loop policy to WindowsSelectorEventLoopPolicy.

---

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Fix NotImplementedError on Windows Python 3.8+
sweep/fix-notimplementederror-windows-python38

Description

This PR fixes the NotImplementedError that occurs when running ProxyBroker from the command line interface (CLI) on Windows with Python 3.8 or higher. The issue is caused by a change in the default event loop policy in Python 3.8 on Windows, which does not support the add_reader method used by ProxyBroker. The fix sets the event loop policy to WindowsSelectorEventLoopPolicy on Windows to resolve the compatibility issue.

Summary of Changes

• Added code to set the event loop policy to WindowsSelectorEventLoopPolicy in the proxybroker/cli.py file.
• The code is placed at the top of the file, after the import statements.
• The fix is only applied on Windows, as the issue does not occur on other platforms.
• The fix ensures that the add_reader method is supported by the event loop policy, resolving the NotImplementedError issue.

---

Step 4: ⌨️ Coding

File	Instructions	Progress
proxybroker/cli.py	Modify proxybroker/cli.py with contents: • At the top of the file, after the import statements, add the following code: python<br/> import sys<br/> import asyncio<br/><br/> if sys.platform == 'win32':<br/> asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())<br/> This code checks if the current platform is Windows ('win32'), and if so, it sets the event loop policy to WindowsSelectorEventLoopPolicy.	✅ Commit 73eb60a

---

Step 5: 🔁 Code Review

Here are my self-reviews of my changes at sweep/fix-notimplementederror-windows-python38.

Here is the 1st review

Thanks for your contribution. There's just a small change needed:

• In proxybroker/cli.py, you've imported sys twice, once at the beginning of the file and again in the added lines (7-12). You only need to import a module once, so you can remove the second import.

Here's how you can update it:

import sys
from contextlib import contextmanager
import asyncio

if sys.platform == 'win32':
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

from . import __version__ as version
from .api import Broker
from .utils import update_geoip_db

Keep up the good work!

I finished incorporating these changes.

---

To recreate the pull request edit the issue title or description.
^{Join Our Discord}

[anon88391]

@bluet The issue doesn't seem fixed with #138 at all. The exception isn't raised anymore, but the whole program seems to be doing absolutely nothing after this change - it just hangs doing literally nothing.
Did you seriously ask GPT4 to fix it and then pushed its code without verifying if it works?

[bluet]

@anon88391 the fix was based on your suggestion.
I don't have Windows machines for decades, and as you do and suggestions that, I applied it.

[bluet]

Seems like aiodns' back in the game
https://github.com/saghul/aiodns/tags

[ziloka]

Seems like aiodns' back in the game
https://github.com/saghul/aiodns/tags

yes, I updated the pull request just to update aiodns and aiohttp dependencies.

just need to fix a unit test that broke

[bluet]

@ziloka I just fixed the error in the unit test and dumped dependencies. Not sure about windows support.