Should we auto reconnect to redis?

[botzill]

I get the following error while the consumer spider running:

RedisMixin.spider_idle of <MySpider 'my_spider' at 0x7f5c2cb335c0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/signal.py", line 32, in send_catch_log
    *arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 121, in spider_idle
    self.schedule_next_requests()
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/spiders.py", line 116, in schedule_next_requests
    self.crawler.engine.crawl(req, spider=self)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 216, in crawl
    self.schedule(request, spider)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/engine.py", line 222, in schedule
    if not self.slot.scheduler.enqueue_request(request):
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/scheduler.py", line 167, in enqueue_request
    self.queue.push(request)
  File "/usr/local/lib/python3.6/dist-packages/scrapy_redis/queue.py", line 104, in push
    self.server.execute_command('ZADD', self.key, score, data)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 730, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 321, in read_response
    raw = self._buffer.readline()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2020-05-01 07:38:27 [scrapy.core.engine] INFO: Closing spider (finished)

I think we need to reconnect to redis if we have cases like this.

[LuckyPigeon]

@botzill
How often does this occurred?

[rmax]

I think makes sense to add some configurable retrying behaviour for known cases.