Fatal MySQL error: Error: read ECONNRESET in Docker logs

[janmarcopetri]

Describe the bug
Since deploying Etherpad (Docker) version 1.9.6 our pads randomly took forever to open (and to create new ones). Reloading the site multiple times fixes it in the end. Not consistent and not directly reproducable. Happens multiple times over the day, not specific to the same pads.
(Something seems to be incorrect with the data sent/received to/from the MariaDB container.)
Docker logs from Etherpad-Container: Fatal MySQL error: Error: read ECONNRESET (see screenshot below)
Docker logs from MariaDB-Container: Aborted connection [...] error reading communication packets (see screenshot below)

To Reproduce
Steps to reproduce the behavior:
Unfortunately unsure for now...
Deploy a Docker Stack with Etherpad 1.9.6 and MariaDB 10.11 ?

Expected behavior
Pads always load and the containers do not produce error logs.

Screenshots

Server (please complete the following information):

• Docker Stack deployed with docker-compose
• Etherpad version: 1.9.6
• MariaDB version: 10.11

[SamTV12345]

Could you please paste your docker-compose anonymized? Maybe I can reproduce it locally.

[janmarcopetri]

Thank you for trying, here you go:

version: '3.8'

volumes:
  mariadb:

services:
    
  mariadb:
    image: registry.<domain>/mirror/mariadb:10.11
    volumes:
      - mariadb:/var/lib/mysql
    environment:
      - MARIADB_ROOT_PASSWORD=
      - MARIADB_DATABASE=etherpad
      - MARIADB_USER=etherpad
      - MARIADB_PASSWORD=
      #- MARIADB_AUTO_UPGRADE=1
      #- MARIADB_INITDB_SKIP_TZINFO=1    
    networks:
      - network
    deploy:
      mode: replicated
      replicas: 1

  etherpad:
    image: registry.<domain>/etherpad:1.9.6
    user: etherpad:etherpad
    environment:
      - TITLE=
      - DEFAULT_PAD_TEXT=
      - ADMIN_USER=admin
      - ADMIN_PASSWORD=
      - DB_TYPE=mysql
      - DB_HOST=mariadb
      - DB_NAME=etherpad
      - DB_USER=etherpad
      - DB_CHARSET=utf8mb4
      - DB_PASS=
      - PAD_OPTIONS_LANG=de
      - PAD_OPTIONS_USE_MONOSPACE_FONT=false
      - SOFFICE=/usr/bin/soffice
      - ABIWORD=/usr/bin/abiword
    networks:
      - network
    ports:
      - "10001:9001"
    deploy:
      mode: replicated
      replicas: 1

networks:
  network:
    driver: overlay

The MariaDB is the official image from Docker Hub, just mirrored on our own registry server.
The Etherpad image is built following the instructions from this repo, adding:

   --build-arg ETHERPAD_PLUGINS="ep_headings2 ep_font_color ep_authornames ep_align" \
    --build-arg INSTALL_ABIWORD=true \
    --build-arg INSTALL_SOFFICE=true

Note we are running a Docker Swarm so probably you will have to remove the "deploy" sections for the containers.

[SamTV12345]

I added some values in the docker-compose.yml and it works so far.

version: '3.8'

volumes:
  mariadb:

services:

  mariadb:
    image: mariadb:10.11
    volumes:
      - mariadb:/var/lib/mysql
    environment:
      - MARIADB_DATABASE=etherpad
      - MARIADB_USER=etherpad
      - MARIADB_PASSWORD=test
      - MARIADB_ROOT_PASSWORD=test
      #- MARIADB_AUTO_UPGRADE=1
      #- MARIADB_INITDB_SKIP_TZINFO=1
    networks:
      - network

  etherpad:
    image: etherpad/etherpad:1.9.6
    user: etherpad:etherpad
    environment:
      - TITLE=
      - DEFAULT_PAD_TEXT=
      - ADMIN_USER=admin
      - ADMIN_PASSWORD=
      - DB_TYPE=mysql
      - DB_HOST=mariadb
      - DB_NAME=etherpad
      - DB_USER=etherpad
      - DB_CHARSET=utf8mb4
      - DB_PASS=test
      - PAD_OPTIONS_LANG=de
      - PAD_OPTIONS_USE_MONOSPACE_FONT=false
      - SOFFICE=/usr/bin/soffice
      - ABIWORD=/usr/bin/abiword
    networks:
      - network
    ports:
      - "10001:9001"

networks:
  network:

[SamTV12345]

I'll let it idle for an hour and get back then. Maybe then I can see errors.

[SamTV12345]

Still worked without a problem. Can you reproduce the error?

[janmarcopetri]

Unfortunately, I myself can't reproduce the error or force it to happen, but it still happens randomly (see new screenshots from today below). It happens with different pads and different users (who then report this to me, but at the time I get to try, it already works again). We are a University with hundreds of different users and multiple pads opened simultaniously.

I've noticed the errors don't get posted at the same time on the database and etherpad logs - might it be possible there are many parallel connections opened to the database and those time out, but etherpad doesn't recognise this and still tries to read/write? Just a wild guess...

More info that might be useful:
We upgraded through multiple versions of etherpad, maybe the errors happened before, but we startet to get reports after the last upgrade from 1.9.3 to 1.9.6.
The MariaDB initially was version 10.7 (I think) and was also upgraded over time to now 10.11.
After getting the reports we "auto-upgraded" the DB with the commented environment vars you saw in my docker-compose. This only fixed some "Incorrect definition of table mysql_column_stats ..." -errors.

[SamTV12345]

Maybe increasing the connections of the MariaDB instance could work? Do you have monitoring enabled for the MariaDB instance? I upgraded to https://www.npmjs.com/package/mysql2. There is also an issue about it: sidorares/node-mysql2#1944