Adding AWS Key & Secret to fluentd s3 plugin

[alexbegg]

Description

The current fluentd.s3.role_arn and fluentd.s3.role_session_name config values (which uses <assume_role_credentials> in the s3 log store) is only possible with Astronomer running in AWS since only something running in AWS can assume an AWS IAM role. If Astronomer is not running in AWS these new config values aws_key_id and aws_sec_key will allow the option of using AWS Key & Secret of an IAM User instead.

See https://github.com/fluent/fluent-plugin-s3/blob/master/docs/credentials.md for fluentd s3 plugin's documentation example of using aws_key_id and aws_sec_key in the plugin.

Related Issues

None

Testing

To test use the following fluentd config with aws_key_id and aws_sec_key in place of role_arn and role_session_name:

fluentd:
  s3:
    enabled: true
    aws_key_id: my-key-id
    aws_sec_key: my-secret-key
    s3_bucket: my-s3-bucket
    s3_region: us-east-1

FYI, in an Astronomer deployment in Azure I have been successfully been using the config value fluentd.extraLogStores to use a custom s3 log store that has with aws_key_id and aws_sec_key in place of the <assume_role_credentials> section like so and it successfully saves log files in the S3 bucket (I did it this way because I did not quite know how to deploy a customized Astronomer chart into my Azure AKS cluster. Once this PR's change is released I will use that instead):

fluentd:
  extraLogStores: |
    <store>
      @type s3
      @id s3
      @log_level info
      aws_key_id MY_AWS_KEY_ID
      aws_sec_key MY_AWS_SECRET_KEY
      s3_bucket currency-airflow-logs
      s3_region us-west-2
      <buffer>
        @type file
        path "/var/log/fluentd-buffers/#{ENV['RELEASE']}-s3-kubernetes.system.buffer"
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
        queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
        overflow_action drop_oldest_chunk
      </buffer>
    </store>

Merging

Do not merge this PR until it lists which release branches this PR should be merged / cherry-picked into.

[danielhoherd]

Thanks for the contribution! I would like to see tests. Here's a test you could crib off of https://github.com/astronomer/astronomer/blob/c1d9b205a6/tests/chart_tests/test_fluentd.py#L106 and here is a tutorial on how tests are written https://github.com/astronomer/astronomer/blob/master/tests/chart_tests/README.md

You can kick off the local pytests with make unittest-charts PYTEST_ADDOPTS='-k fluentd' @ me if you need help.