How to migrate to OCI

[funes79]

I would like to upgrade Karpenter app but the latest available version seems to be in OCI registry. According to docs (https://karpenter.sh/v0.18.1/getting-started/getting-started-with-terraform/) the OCI url should be used. However in ArgoCD the OCI url is not recognized:

rpc error: code = Unknown desc = `helm pull --destination /tmp/5c46eac2-2448-48e2-bbe0-814efae2f904 --version 0.18.1 --repo oci://public.ecr.aws/karpenter karpenter` failed exit status 1: Error: looks like "oci://public.ecr.aws/karpenter" is not a valid chart repository or cannot be reached: object required

So what should I use in the Helm chart definition for Karpenter?

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: karpenter-eks
  namespace: argocd
spec:
  destination:
    namespace: karpenter
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: karpenter
    repoURL: oci://public.ecr.aws/karpenter
    # #https://charts.karpenter.sh/
    targetRevision: v0.18.1

It would be good to have a clear docs about how to proceed with Helm applications defined in public AWS or Azure registries.
Thanks

[funes79]

Tried the URL without oci but it seems not working.

With:

spec:
  destination:
    namespace: karpenter
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: karpenter
    repoURL: public.ecr.aws

the error is:

rpc error: code = Unknown desc = `helm pull oci://public.ecr.aws/karpenter --version 0.18.1 --destination /tmp/cd5f0caf-4507-4a3b-875c-4653f3dd870d` failed exit status 1: Error: public.ecr.aws/karpenter:0.18.1: not found

spec:
  destination:
    namespace: karpenter
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: karpenter
    repoURL: public.ecr.aws/karpenter
    # oci://public.ecr.aws/karpenter/karpenter 
    targetRevision: 0.18.1

Another attempt:

rpc error: code = Unknown desc = `helm pull oci://public.ecr.aws/karpenter/karpenter --version 0.18.1 --destination /tmp/517a6ff7-5b73-4cc5-8d2d-a5a43be304e2` failed exit status 1: Error: public.ecr.aws/karpenter/karpenter:0.18.1: not found

also tried

spec:
  destination:
    namespace: karpenter
    server: https://kubernetes.default.svc
  project: default
  source:
    chart: karpenter
    repoURL: public.ecr.aws/karpenter/karpenter
    # oci://public.ecr.aws/karpenter/karpenter 
    targetRevision: 0.18.1

similar error

ComparisonError: rpc error: code = Unknown desc = `helm pull oci://public.ecr.aws/karpenter/karpenter --version 0.18.1 --destination /tmp/3250ec2d-cc14-4ed7-8f2d-2447aa470026` failed exit status 1: Error: public.ecr.aws/karpenter/karpenter:0.18.1: not found/tmp/6b65e11a-4259-4b13-a175-dae1e573a1b1` failed exit status 1: Error: public.ecr.aws/karpenter/karpenter/karpenter:0.18.1: not found

[blackliner]

Trying to do the same, no luck so far. Does argocd support OCI helm charts at all?

[kamirendawkins]

We have the same issue and this is what I have sorted out so far:

Public OCI Repos, such as Karpenter's Public ECR Repository, do not appear to work as expected. If I recall it has to do with the pull command used but I will see if I can find that information and update.

Does argocd support OCI helm charts at all?

Yes, we use this for internal ECR Projects that maintain Helm Charts all the time. The gotcha here is that since they are in ECR we have to maintain expiring ECR Credentials which is annoyingly solved with Argo-Workflows and a pre-sync call to force update the repository credentials. This could be done on cron to reduce frequent API calls but it doesn't help in any way when it is a public ECR Repo.

Knowing this I tried to leverage ECRs Pull-Through Cache...thinking that since it allows for a Private ECR Endpoint for the Public Gallery this should work fine. That turned into a bug for AWS: aws/containers-roadmap#1855 so that also does not work.

I know this does not help but maybe it will add some flavor so we can sort this out.

Issues worth Tracking:

• Helm OCI Registry with refresh token #8097
• Using oci:// prefix in repository errors with not valid chart error #10823

Adjacent:

• feat: Add wildcard support in OCI Helm Repositories targetRevision (#6686) #10641

[blackliner]

Ok, I played a bit more around, and got ArgoCD to deploy karpenter v0.18.1 through an ApplicationSet:
karpenter-app.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: karpenter
  namespace: infra-argo-cd
spec:
  generators:
  - list:
      elements:
      - cluster: "in-cluster"
        url: "https://kubernetes.default.svc"
  template:
    metadata:
      name: "karpenter"
    spec:
      project: infra
      source:
        # ArgoCD will create a pull command by combining {source.repoURL}/{source.chart}
        repoURL: public.ecr.aws/karpenter
        targetRevision: "v0.18.1"
        chart: karpenter
        path: ""
        helm:
          version: v3
          releaseName: karpenter
          values: |
            skipped
      destination:
        server: '{{ url }}'
        namespace: "infra-karpenter"
      syncPolicy:
        automated:
          selfHeal: true
          prune: false
        syncOptions:
        - CreateNamespace=true

argocd.yaml

...
            configs:
              repositories:
                karpenter-helm-repo:
                  url: https://charts.karpenter.sh
                  type: helm
                  name: karpenter
                karpenter-oci-helm-repo:
                  url: public.ecr.aws/karpenter/karpenter
                  type: helm
                  name: karpenter
                  enableOCI: "true"
...

[rarecrumb]

Why is an ApplicationSet necessary for this?

[satishweb]

ApplicationSet is used to deploy same application to multiple clusters using cluster labels.

[FelipeAllure]

any updates on OCI support?
currently on v2.7.0 and still the URL oci://public.ecr.aws/karpenter is not being recognized

[kubical-moncef]

Alternatively, you can make use of the path param when defining the argocd application, and use the official karpenter github repository like so:

kind: Application
metadata:
  name: karpenter
  namespace: argocd
spec:
  project: infrastructure
  source:
    # Use the official aws karpenter repoUrl
    repoURL: https://github.com/aws/karpenter
    # Adjust the path
    path: charts/karpenter
    # Choose release. Do not use HEAD if auto-sync if enabled
    targetRevision: release-v0.28.1
    helm:
      values: |
        serviceAccount:
          annotations: 
            eks.amazonaws.com/role-arn: ${karpenter_iam_role_arn}
        settings:
          aws:
            clusterName: ${cluster_name}
            clusterEndpoint: ${cluster_endpoint}
            defaultInstanceProfile: KarpenterNodeInstanceProfile-${cluster_name}
            interruptionQueueName: ${cluster_name}
  destination:
    server: https://kubernetes.default.svc
    namespace: ${namespace}
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      selfHeal: true
      prune: true

Hope this helps

[Uritassa]

this one worked for me. thanks

[elmariofredo]

For me it's working without any issue.

This is my example for dragonfly installation

Based on install guide:

helm upgrade --install dragonfly oci://ghcr.io/dragonflydb/dragonfly/helm/dragonfly --version v1.6.1

I created Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dragonfly
  namespace: argo-cd-system
spec:
  project: core
  source:
    chart: dragonfly
    repoURL: ghcr.io/dragonflydb/dragonfly/helm
    targetRevision: v1.6.1
    helm:
      releaseName: main
      values: |
        replicaCount: 1
        storage:
          enabled: true
          requests: 512Mi

        podSecurityContext:
          fsGroup: 2000

        securityContext:
          capabilities:
            drop:
              - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000

  destination:
    server: https://kubernetes.default.svc
    namespace: core-prod
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

ArgoCD will deploy dragonfly from OCI source without any other setup 🎉

[trojaond]

I have made Karpenter work with ArgoCD Application (ArgoCD 2.9.3):

1. You need to configure a repository: (in my case configure argocd-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repositories:
    karpenter-oci:
      url: public.ecr.aws/karpenter/karpenter
      type: helm
      name: karpenter
      enableOCI: "true"

2. Create ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: karpenter
  namespace: argocd
  # !!! cascade delete !!!
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: xxx
  destination:
    namespace: kube-system
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: public.ecr.aws/karpenter
    targetRevision: "v0.33.1"
    chart: karpenter
    helm:
      values: |-
        ...
  syncPolicy:
    automated: 
      prune: true
      selfHeal: true