Spec for meltano config support for encryption

[edgarrmondragon]

Part of a multi-part set of feature releases:

1. Spec for `meltano config` support for encryption #6999 (this item)
   • 1. feature: Support for Encrypted Config Values in meltano config #6987
2. Spec discussion: Meltano Cloud as an alternative config store backends #7055
3. Spec discussion: allow pointers in yaml to reference settings values in other stores #7056

---

The CLI will support encryption of config values.

The following command will encrypt <config value>

meltano config <plugin name> set --secret <config name> <config value>

Automatic secret detection

All settings of kind password will automatically encrypted when set in the CLI, if there's a default_encryption_key:

$ meltano config <a plugin> set <a password setting> <a secret>

Storing encrypted secrets

Secrets will be stored in a structured string similar to Sops's notation:

ENC(name=kms,uri='awskms://1234abcd-12ab-34cd-56ef-1234567890ab?region=us-east-1',cipher=...)

The exact format of this string is still TBD, but it would at least encode:

• The name of the key used for encryption
• The uri of the key needed for decrytion
• The secret ciphertext

Secrets (their ciphertext) are stored along with metadata that helps Meltano identify the decryption key.

plugins:
  extractors:
  - name: <a plugin>
    config:
      <a password setting>: 'ENC(name=kms,uri='awskms://1234abcd-12ab-34cd-56ef-1234567890ab?region=us-east-1',cipher=...)'

The original encryption key is stored inline to ensure the secret can always be decrypted, even if keys are removed from the project at some point.

Encryption and Decryption

1. All keys are identified by a unique ID or name presented.
2. There's always ONE AND ONLY ONE encryption key (EK) enabled in a project, referenced by name as in secrets_config.meltano-cloud.
3. There's ONE OR MORE decryption keys (DK) in the secrets_config.keys[] array.

The corresponding ciphertext will be processed by

• a key managed by Meltano Cloud

• A bring-your-own key with initial support for AWA KMS, and with plans to support HashiCorp Vault and GCP KMS:

  secrets_config:
    keys:
    - name: kms
      uri: awskms://1234abcd-12ab-34cd-56ef-1234567890ab?region=us-east-1
    - name: vault
      uri: hashivault://mykey 
    - name: meltano-cloud
      uri: meltano://project-id
    encryption_key: meltano-cloud
    default_store: dotenv  # not needed for V1

Key rotation

Not supported in V1.

Changelog:

• 2022-11-18: Changed secrets_provider to default_encryption_key

cc @aaronsteers

[aaronsteers]

@edgarrmondragon - Do we have a logged issue for this? Do you mind linking in the description if so?

[edgarrmondragon]

@aaronsteers done :)