[kube-prometheus-stack] Grafana Not Starting Database is Locked Error

[marvagabi]

Describe the bug a clear and concise description of what the bug is.

Configured both Prometheus and Grafana for persistent storage with an established storage class within my environment. Prometheus is working fine with the storage class. Grafana on the other hand is having several issues, it is always giving a "database is locked" error on startup.

I've done a lot of reading about this same issue with other reports and have tried various fixes such as setting WAL to true or cache_mode to shared. I even disabled migration locking as initially the error would occur after the pod started trying to migrate the db.

logger=settings t=2024-05-15T20:51:39.987644051Z level=info msg="Starting Grafana" version=10.4.1 commit=d94d597d847c05085542c29dfad6b3f469cc77e1 branch=v10.4.x compiled=2024-05-15T20:51:39Z logger=settings t=2024-05-15T20:51:39.988028098Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini logger=settings t=2024-05-15T20:51:39.988064088Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini logger=settings t=2024-05-15T20:51:39.988071518Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana/" logger=settings t=2024-05-15T20:51:39.988077535Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" logger=settings t=2024-05-15T20:51:39.988083581Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" logger=settings t=2024-05-15T20:51:39.988089397Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" logger=settings t=2024-05-15T20:51:39.988095214Z level=info msg="Config overridden from command line" arg="default.log.mode=console" logger=settings t=2024-05-15T20:51:39.988101498Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana/" logger=settings t=2024-05-15T20:51:39.988116457Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" logger=settings t=2024-05-15T20:51:39.988122581Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" logger=settings t=2024-05-15T20:51:39.988132678Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" logger=settings t=2024-05-15T20:51:39.988138701Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=k8smonitor" logger=settings t=2024-05-15T20:51:39.988144637Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********" logger=settings t=2024-05-15T20:51:39.988162304Z level=info msg=Target target=[all] logger=settings t=2024-05-15T20:51:39.988173144Z level=info msg="Path Home" path=/usr/share/grafana logger=settings t=2024-05-15T20:51:39.988178983Z level=info msg="Path Data" path=/var/lib/grafana/ logger=settings t=2024-05-15T20:51:39.988184649Z level=info msg="Path Logs" path=/var/log/grafana logger=settings t=2024-05-15T20:51:39.988190159Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins logger=settings t=2024-05-15T20:51:39.988195875Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning logger=settings t=2024-05-15T20:51:39.988201663Z level=info msg="App mode production" logger=sqlstore t=2024-05-15T20:51:39.988738468Z level=info msg="Connecting to DB" dbtype=sqlite3 logger=sqlstore t=2024-05-15T20:51:39.988793637Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxrwxrwx expected=-rw-r----- logger=migrator t=2024-05-15T20:51:39.989905788Z level=info msg="Starting DB migrations" Error: ✗ failed to check table existence: database is locked

What's your helm version?

3.14.2

What's your kubectl version?

v1.29.4

Which chart?

kube-prometheus-stack-58.4.0

What's the chart version?

58.4.0

What happened?

Configured Grafana for persistent storage utilizing a storage class within my kuberentes cluster.

What you expected to happen?

Grafana to startup with persistent storage enabled.

How to reproduce it?

Add settings for persistent storage in values.yaml file and upgrade helm chart.

Enter the changed values of values.yaml?

grafana:
grafana.ini:
database:
wal: true
cache_mode: shared
migration_locking: false
enabled: true
fullnameOverride: grafana
forceDeployDatasources: false
forceDeployDashboards: false
defaultDashboardsEnabled: true
defaultDashboardsTimezone: edt
serviceMonitor:
enabled: true
admin:
existingSecret: grafana-admin-credentials
userKey: admin-user
passwordKey: admin-password
service:
type: NodePort
port: 3000
targetPort: 3000
portName: grafana-https
persistence:
type: pvc
enabled: true
storageClassName: smb
accessModes:
- ReadWriteOnce
size: 10Gi
finalizers:
- kubernetes.io/pvc-protection

Enter the command that you execute and failing/misfunctioning.

helm upgrade -n monitoring k8s-mon prometheus-community/kube-prometheus-stack -f /helm-values-dev.yaml

Anything else we need to know?

No response