Unhandled patched exception MySQL server has gone away

[mpw1337]

This night my scanner stopped completely, because the connection to my DB got lost.

Expected Behavior

Catch the exception, when the connection to DB get's lost and try again after some seconds/minutes

Current Behavior

Exception happens and scanner get's stopped, but process continues to run.

2017-12-10 04:12:05,779 [   search-overseer][     runserver][CRITICAL] Unhandled patched exception (<class 'pymysql.err.OperationalError'>): "(2006, "MySQL server has gone away (error(32, 'Broken pipe'))")".
2017-12-10 04:12:05,781 [   search-overseer][     runserver][   ERROR] Uncaught exception
Traceback (most recent call last):
  File "./runserver.py", line 90, in run
    run_old(*args, **kwargs)
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/alarm/RocketMap/pogom/search.py", line 487, in search_overseer_thread
    upsertKeys(args.hash_key, key_scheduler, db_updates_queue)
  File "/home/alarm/RocketMap/pogom/search.py", line 1213, in upsertKeys
    HashKeys.getStoredPeak(key))
  File "/home/alarm/RocketMap/pogom/models.py", line 1833, in getStoredPeak
    return result
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4022, in __exit__
    self.txn.__exit__(exc_type, exc_val, exc_tb)
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4091, in __exit__
    self.rollback(False)
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4071, in rollback
    self.db.rollback()
  File "/usr/lib/python2.7/site-packages/peewee.py", line 3502, in rollback
    self.get_conn().rollback()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
    self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1056, in _execute_command
    self._write_bytes(packet)
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1008, in _write_bytes
    raise err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
OperationalError: (2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
2017-12-10 04:12:05,935 [  search-worker-18][        search][   ERROR] Map parse failed at 49.471478,8.152250, abandoning location. Hypnoquappe5009 may be banned.. Exception message: OperationalError(2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
Traceback (most recent call last):
  File "/home/alarm/RocketMap/pogom/search.py", line 1009, in search_worker_thread
    scan_date, account, account_sets)
  File "/home/alarm/RocketMap/pogom/models.py", line 1994, in parse_map
    sighting)
  File "/home/alarm/RocketMap/pogom/models.py", line 1530, in classify
    .order_by(SpawnpointDetectionData.scan_time.asc()).dicts())
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4022, in __exit__
    self.txn.__exit__(exc_type, exc_val, exc_tb)
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4086, in __exit__
    self.rollback(False)
  File "/usr/lib/python2.7/site-packages/peewee.py", line 4071, in rollback
    self.db.rollback()
  File "/usr/lib/python2.7/site-packages/peewee.py", line 3502, in rollback
    self.get_conn().rollback()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
    self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1056, in _execute_command
    self._write_bytes(packet)
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1008, in _write_bytes
    raise err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
OperationalError: (2006, "MySQL server has gone away (error(32, 'Broken pipe'))")

Possible Solution

Stop process completely or handle the exception with a timeout.

Steps to Reproduce (for bugs)

Don't know if that really works, because it's the first time I got this bug.

1. Start RM
2. Disconnect Internet Connection, when Database is on an external server

Context

Your Environment

• Version used:
• Environment name: Python 2.7.14
• Operating System: Arch Linux ARM on Rpi3
• Database: External MySQL Database

[FrostTheFox]

Not particularly a bug, but it definitely is an exception that should be handled.

[commitBlob]

@mpw1337 Check how many connections to db you have, if it is over max-connections parameter you will have above issue.
We restarted msyql and it is working fine now.
It is not a fix to the issue but it keeps map running.

[farstarss]

This issue still seems to persist. When I stop mysql, wait a few seconds and start it again RocketMap does not seem to be able to recover from this. Can't exactly tell you right now if it's the same Exception but I've experience this a few times now and the only solution so far was restarting the RM instances.

[FrostTheFox]

Could still exist.

[vamsilella]

restarting Mysql is a temporary solution instead you can check the wait_timeout mysql variable in your my.cnf configuration file is large enough also increase the maximal packet size limit for max_allowed_packet