How to disable Prometheus and Grafana? #502

carolili · 2021-03-05T15:21:08Z

carolili
Mar 5, 2021

Hello, thanks for creating this project! I'm trying to deploy k8ssandra without Prometheus and Grafana (as it seems to conflict with our existing kube-prometheus-stack installation). However setting the below values in the chart to false does not seem to help. Grafana and Prometheus are still attempting to deploy. Am I missing something? Thanks in advance for any advice.

Chart version: 1.0.0

monitoring.grafana.provision_dashboards
monitoring.prometheus.provision_service_monitors
kube-prometheus-stack.enabled
kube-prometheus-stack.prometheusOperator.enabled
kube-prometheus-stack.prometheus.enabled
kube-prometheus-stack.grafana.enabled

Answered by jsanda

Mar 5, 2021

Hi @carolili

I generally use an overrides file as follows,

$ helm install dev k8ssandra/k8ssandra -f my-values.yaml

and my-values.yaml should include this:

kube-prometheus-stack:
  enabled: false

This will prevent the kube-prometheus-stack chart from getting installed.

View full answer

jsanda · 2021-03-05T17:14:43Z

jsanda
Mar 5, 2021
Maintainer

Hi @carolili

I generally use an overrides file as follows,

$ helm install dev k8ssandra/k8ssandra -f my-values.yaml

and my-values.yaml should include this:

kube-prometheus-stack:
  enabled: false

This will prevent the kube-prometheus-stack chart from getting installed.

1 reply

carolili Mar 7, 2021
Author

Hi @jsanda ! Thank you for the response. Could you confirm that the setting you mention also works with the k8ssandra chart 1.0.0? When I run:

helm --dry-run --debug . --generate-name -f my-values.yaml

then I still see Prometheus configuration being generated. I would expect that with enabled: false the manifests would not contain any reference to kube-prometheus-stack.

I tried using the same value as you mention in my-values.yaml with no other option added.

$ cat my-values.yaml
kube-prometheus-stack:
  enabled: false

I also tried installing k8ssandra with the option you mentioned, to check if there is any difference compared to using --dry-run, but Prometheus still gets deployed 🤔

helm install -n k8ssandra-test . --generate-name -f my-values.yaml

Any ideas are welcome. As mentioned in my first message, I've tried using all six configuration values related to Prometheus/Grafana but they don't seem to disable the installation.

jsanda · 2021-03-07T22:38:28Z

jsanda
Mar 7, 2021
Maintainer

Could you confirm that the setting you mention also works with the k8ssandra chart 1.0.0?

Yes they do work with 1.0.0.

Observe the output when I disable it:

Johns-MacBook-Pro:charts jsanda$ helm template test k8ssandra-stable/k8ssandra --set kube-prometheus-stack.enabled=false | grep kube-prometheus
Johns-MacBook-Pro:charts jsanda$

vs when I enable it:

Johns-MacBook-Pro:charts jsanda$ helm template test k8ssandra-stable/k8ssandra --set kube-prometheus-stack.enabled=true | grep kube-prometheus
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/podsecuritypolicy.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-podsecuritypolicy.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/psp.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/psp.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/serviceaccount.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-serviceaccount.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/serviceaccount.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/serviceaccount.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/secret.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/configmap-dashboard-provider.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/configmap.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-configmap.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/grafana/configmaps-datasources.yaml
  name: test-kube-prometheus-stack-grafana-datasource
    app: kube-prometheus-stack-grafana
    chart: kube-prometheus-stack-12.11.3
      url: http://test-kube-prometheus-stack-prometheus:9090/
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/clusterrole.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/clusterrole.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/psp-clusterrole.yaml
  name: test-kube-prometheus-stack-operator-psp
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
  - test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/clusterrole.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
# This permission are not in the kube-prometheus repo
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/psp-clusterrole.yaml
  name: test-kube-prometheus-stack-prometheus-psp
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
  - test-kube-prometheus-stack-prometheus
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/clusterrolebinding.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/clusterrolebinding.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-operator
  name: test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/psp-clusterrolebinding.yaml
  name: test-kube-prometheus-stack-operator-psp
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-operator-psp
    name: test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/clusterrolebinding.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-prometheus
    name: test-kube-prometheus-stack-prometheus
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/psp-clusterrolebinding.yaml
  name: test-kube-prometheus-stack-prometheus-psp
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-prometheus-psp
    name: test-kube-prometheus-stack-prometheus
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/role.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-role.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/rolebinding.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test-rolebinding.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/service.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/service.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
    app: kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/service.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
    prometheus: test-kube-prometheus-stack-prometheus
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/deployment.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/deployment.yaml
  name: test-kube-prometheus-stack-operator
    app: kube-prometheus-stack-operator
    chart: kube-prometheus-stack-12.11.3
      app: kube-prometheus-stack-operator
        app: kube-prometheus-stack-operator
        chart: kube-prometheus-stack-12.11.3
        - name: kube-prometheus-stack
            - --kubelet-service=kube-system/test-kube-prometheus-stack-kubelet
            secretName: test-kube-prometheus-stack-admission
      serviceAccountName: test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/mutatingWebhookConfiguration.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
        name: test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/prometheus.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack-prometheus
    chart: kube-prometheus-stack-12.11.3
  externalUrl: http://test-kube-prometheus-stack-prometheus.default:9090
  serviceAccountName: test-kube-prometheus-stack-prometheus
      app: kube-prometheus-stack
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/alertmanager.rules.yaml
  name: test-kube-prometheus-stack-alertmanager.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
      expr: count by(namespace,service) (count_values by(namespace,service) ("config_hash", alertmanager_config_hash{job="test-kube-prometheus-stack-alertmanager",namespace="default"})) != 1
      expr: alertmanager_config_last_reload_successful{job="test-kube-prometheus-stack-alertmanager",namespace="default"} == 0
        alertmanager_cluster_members{job="test-kube-prometheus-stack-alertmanager",namespace="default"}
        count by (service) (alertmanager_cluster_members{job="test-kube-prometheus-stack-alertmanager",namespace="default"})
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/general.rules.yaml
  name: test-kube-prometheus-stack-general.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/k8s.rules.yaml
  name: test-kube-prometheus-stack-k8s.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-prometheus-general.rules.yaml
  name: test-kube-prometheus-stack-kube-prometheus-general.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
  - name: kube-prometheus-general.rules
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-prometheus-node-recording.rules.yaml
  name: test-kube-prometheus-stack-kube-prometheus-node-recording.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
  - name: kube-prometheus-node-recording.rules
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kube-state-metrics.yaml
  name: test-kube-prometheus-stack-kube-state-metrics
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-apps.yaml
  name: test-kube-prometheus-stack-kubernetes-apps
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-resources.yaml
  name: test-kube-prometheus-stack-kubernetes-resources
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-storage.yaml
  name: test-kube-prometheus-stack-kubernetes-storage
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-apiserver.yaml
  name: test-kube-prometheus-stack-kubernetes-system-apiserver
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system-kubelet.yaml
  name: test-kube-prometheus-stack-kubernetes-system-kubelet
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-system.yaml
  name: test-kube-prometheus-stack-kubernetes-system
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-exporter.rules.yaml
  name: test-kube-prometheus-stack-node-exporter.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-exporter.yaml
  name: test-kube-prometheus-stack-node-exporter
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node-network.yaml
  name: test-kube-prometheus-stack-node-network
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/node.rules.yaml
  name: test-kube-prometheus-stack-node.rules
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/prometheus-operator.yaml
  name: test-kube-prometheus-stack-prometheus-operator
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
      expr: (sum by (controller,namespace) (rate(prometheus_operator_list_operations_failed_total{job="test-kube-prometheus-stack-operator",namespace="default"}[10m])) / sum by (controller,namespace) (rate(prometheus_operator_list_operations_total{job="test-kube-prometheus-stack-operator",namespace="default"}[10m]))) > 0.4
      expr: (sum by (controller,namespace) (rate(prometheus_operator_watch_operations_failed_total{job="test-kube-prometheus-stack-operator",namespace="default"}[10m])) / sum by (controller,namespace) (rate(prometheus_operator_watch_operations_total{job="test-kube-prometheus-stack-operator",namespace="default"}[10m]))) > 0.4
      expr: min_over_time(prometheus_operator_syncs{status="failed",job="test-kube-prometheus-stack-operator",namespace="default"}[5m]) > 0
      expr: (sum by (controller,namespace) (rate(prometheus_operator_reconcile_errors_total{job="test-kube-prometheus-stack-operator",namespace="default"}[5m]))) / (sum by (controller,namespace) (rate(prometheus_operator_reconcile_operations_total{job="test-kube-prometheus-stack-operator",namespace="default"}[5m]))) > 0.1
      expr: rate(prometheus_operator_node_address_lookup_errors_total{job="test-kube-prometheus-stack-operator",namespace="default"}[5m]) > 0.1
      expr: min by(namespace, controller) (max_over_time(prometheus_operator_ready{job="test-kube-prometheus-stack-operator",namespace="default"}[5m]) == 0)
      expr: min_over_time(prometheus_operator_managed_resources{state="rejected",job="test-kube-prometheus-stack-operator",namespace="default"}[5m]) > 0
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/prometheus.yaml
  name: test-kube-prometheus-stack-prometheus
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-12.11.3
        max_over_time(prometheus_config_last_reload_successful{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) == 0
          predict_linear(prometheus_notifications_queue_length{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m], 60 * 30)
          min_over_time(prometheus_notifications_queue_capacity{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          rate(prometheus_notifications_errors_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          rate(prometheus_notifications_sent_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          rate(prometheus_notifications_errors_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          rate(prometheus_notifications_sent_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
        max_over_time(prometheus_notifications_alertmanagers_discovered{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) < 1
      expr: increase(prometheus_tsdb_reloads_failures_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[3h]) > 0
      expr: increase(prometheus_tsdb_compactions_failed_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[3h]) > 0
      expr: rate(prometheus_tsdb_head_samples_appended_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) <= 0
      expr: rate(prometheus_target_scrapes_sample_duplicate_timestamp_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) > 0
      expr: rate(prometheus_target_scrapes_sample_out_of_order_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) > 0
          rate(prometheus_remote_storage_failed_samples_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
            rate(prometheus_remote_storage_failed_samples_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
            rate(prometheus_remote_storage_succeeded_samples_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          max_over_time(prometheus_remote_storage_highest_timestamp_in_seconds{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          max_over_time(prometheus_remote_storage_queue_highest_sent_timestamp_seconds{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
        description: Prometheus {{$labels.namespace}}/{{$labels.pod}} remote write desired shards calculation wants to run {{ $value }} shards for queue {{ $labels.remote_name}}:{{ $labels.url }}, which is more than the max of {{ printf `prometheus_remote_storage_shards_max{instance="%s",job="test-kube-prometheus-stack-prometheus",namespace="default"}` $labels.instance | query | first | value }}.
          max_over_time(prometheus_remote_storage_shards_desired{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
          max_over_time(prometheus_remote_storage_shards_max{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m])
      expr: increase(prometheus_rule_evaluation_failures_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) > 0
      expr: increase(prometheus_rule_group_iterations_missed_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) > 0
      expr: increase(prometheus_target_scrape_pool_exceeded_target_limit_total{job="test-kube-prometheus-stack-prometheus",namespace="default"}[5m]) > 0
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/validatingWebhookConfiguration.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
        name: test-kube-prometheus-stack-operator
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/psp.yaml
  name: test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/serviceaccount.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrole.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
    - test-kube-prometheus-stack-admission
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/clusterrolebinding.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-admission
    name: test-kube-prometheus-stack-admission
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/rolebinding.yaml
  name:  test-kube-prometheus-stack-admission
    app: kube-prometheus-stack-admission
    chart: kube-prometheus-stack-12.11.3
  name: test-kube-prometheus-stack-admission
    name: test-kube-prometheus-stack-admission
# Source: k8ssandra/charts/kube-prometheus-stack/charts/grafana/templates/tests/test.yaml
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/job-createSecret.yaml
  name:  test-kube-prometheus-stack-admission-create
    app: kube-prometheus-stack-admission-create
    chart: kube-prometheus-stack-12.11.3
      name:  test-kube-prometheus-stack-admission-create
        app: kube-prometheus-stack-admission-create
        chart: kube-prometheus-stack-12.11.3
            - --host=test-kube-prometheus-stack-operator,test-kube-prometheus-stack-operator.default.svc
            - --secret-name=test-kube-prometheus-stack-admission
      serviceAccountName: test-kube-prometheus-stack-admission
# Source: k8ssandra/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/job-patchWebhook.yaml
  name:  test-kube-prometheus-stack-admission-patch
    app: kube-prometheus-stack-admission-patch
    chart: kube-prometheus-stack-12.11.3
      name:  test-kube-prometheus-stack-admission-patch
        app: kube-prometheus-stack-admission-patch
        chart: kube-prometheus-stack-12.11.3
            - --webhook-name=test-kube-prometheus-stack-admission
            - --secret-name=test-kube-prometheus-stack-admission
      serviceAccountName: test-kube-prometheus-stack-admission
Johns-MacBook-Pro:charts jsanda$

Note that when kube-prometheus-stack is disabled we still create a ServiceMonitor as well as the ConfigMap(s) for the Grafana dashboards.

0 replies

carolili · 2021-03-08T08:38:12Z

carolili
Mar 8, 2021
Author

Hi, thanks for the response. I see the same with helm template test .... Could there be a difference in helm template test ... and helm install ...? I tried the following:

$ helm install --dry-run --debug k8ssandra/k8ssandra --generate-name --set kube-prometheus-stack.enabled=false | grep prometheus

install.go:172: [debug] Original chart version: ""
install.go:189: [debug] CHART PATH: /Users/carolili/Library/Caches/helm/repository/k8ssandra-1.0.0.tgz

install.go:189: [debug] WARNING: This chart or one of its subcharts contains CRDs. Rendering may fail or contain inaccuracies.
kube-prometheus-stack:
kube-prometheus-stack:
        repository: quay.io/prometheus/alertmanager
  prometheus:
        prefix: prometheus
    prometheusSpec:
        repository: quay.io/prometheus/prometheus
      prometheusExternalLabelName: ""
      prometheusExternalLabelNameClear: false
      prometheusRulesExcludedFromEnforce: false
  prometheusOperator:
      repository: quay.io/prometheus-operator/prometheus-operator
    prometheusConfigReloaderImage:
      repository: quay.io/prometheus-operator/prometheus-config-reloader
    prometheusInstanceNamespaces: []
  prometheus:
                "query": "prometheus",
                "query": "prometheus",
                "query": "prometheus",
# Source: k8ssandra/templates/prometheus/service_monitor.yaml
  name: k8ssandra-1615192285-prometheus-k8ssandra
  - port: prometheus
  name: k8ssandra-1615192285-prometheus-dc1-stargate

1 reply

jsanda Mar 9, 2021
Maintainer

I have done some more testing and having trouble reproducing your output. Here is a summary of my findings:

With kube-prometheus-stack.enabled: false I do not any objects from kube-prometheus-stack created except for CRDs.
I stumbled onto CRD's from subchart being deployed even when subchart dependencies.condition is false helm/helm#8508. Even when a subchart is disabled, its CRDs will get deployed.
Even when kube-prometheus-stack is disabled k8ssandra creates a couple ServiceMonitors. They can be disabled with monitoring.prometheus.provision_service_monitors: false.
helm install --dry-run does not install CRDs so it produce some errors if your install needs to deploy CRDs.

@carolili can you please try and install with kube-prometheus-stack.enabled: false and --debug. The output will show the templates that get rendered. My expectation is that nothing from kube-prometheus-stack is rendered with the exception of CRDs if they are not already installed.

carolili · 2021-03-11T12:39:28Z

carolili
Mar 11, 2021
Author

Thank you for the debugging and advice 🙂 I ended up reinstalling k8ssandra and having some trouble to delete resources (the Cassandradatacenters CRD), so I installed the helm chart in another namespace and this time the kube-prometheus-stack.enabled: false setting works.

I'm not sure what the issue was. Perhaps I had a misconfiguration the first time I installed the chart and it remained, causing Prometheus and Grafana to be installed even if I kept redeploying it.

1 reply

jsanda Mar 11, 2021
Maintainer

Excellent! Thanks for the update.

I ended up reinstalling k8ssandra and having some trouble to delete resources (the Cassandradatacenters CRD)

Helm takes a bit of a hands off approach with CRDs. It will install them if they are not already installed. It does not doing anything beyond that. It will not delete them on helm uninstall for example.

If deleting the CassandraDatacenter CRD gets stuck in the future you need to clear the finalizer as follows:

kubectl patch cassandradatacenters.cassandra.datastax.com -p '{"metadata":{"finalizers":[]}}' --type=merge

That will unblock it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to disable Prometheus and Grafana? #502

{{title}}

Replies: 4 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to disable Prometheus and Grafana? #502

carolili Mar 5, 2021

Replies: 4 comments · 3 replies

jsanda Mar 5, 2021 Maintainer

carolili Mar 7, 2021 Author

jsanda Mar 7, 2021 Maintainer

carolili Mar 8, 2021 Author

jsanda Mar 9, 2021 Maintainer

carolili Mar 11, 2021 Author

jsanda Mar 11, 2021 Maintainer

carolili
Mar 5, 2021

Replies: 4 comments 3 replies

jsanda
Mar 5, 2021
Maintainer

carolili Mar 7, 2021
Author

jsanda
Mar 7, 2021
Maintainer

carolili
Mar 8, 2021
Author

jsanda Mar 9, 2021
Maintainer

carolili
Mar 11, 2021
Author

jsanda Mar 11, 2021
Maintainer