AWS Kinesis Firehose Terraform module

Dynamic Terraform module, which creates a Kinesis Firehose Stream and others resources like Cloudwatch, IAM Roles and Security Groups that integrate with Kinesis Firehose. Supports all destinations and all Kinesis Firehose Features.

Table of Contents

• Table of Contents
• Module versioning rule
• Features
• How to Use
  • Sources
    • Kinesis Data Stream
      • Kinesis Data Stream Encrypted
    • Direct Put
    • WAF
    • MSK
  • Destinations
    • S3
    • Redshift
    • Elasticsearch
    • Opensearch
    • Opensearch Serverless
    • Splunk
    • HTTP Endpoint
    • Datadog
    • New Relic
    • Coralogix
    • Dynatrace
    • Honeycomb
    • Logic Monitor
    • MongoDB
    • SumoLogic
  • Server Side Encryption
  • Data Transformation with Lambda
  • Data Format Conversion
  • Dynamic Partition
  • S3 Backup Data
  • Destination Delivery Logging
  • VPC Support
    • ElasticSearch / Opensearch / Opensearch Serverless
    • Redshift / Splunk
  • Application Role
• Destinations Mapping
• Examples
• Requirements
• Providers
• Modules
• Resources
• Inputs
• Outputs
• Upgrade
• Deprecations
  • Version 3.1.0
  • Version 3.3.0
• License

Module versioning rule

Module version	AWS Provider version
>= 1.x.x	~> 4.4
>= 2.x.x	~> 5.0
>= 3.x.x	>= 5.33

Features

• Sources
  • Kinesis Data Stream
  • Direct Put
  • WAF
  • MSK
• Destinations
  • S3
    • Data Format Conversion
    • Dynamic Partition
  • Redshift
    • VPC Support. Security Groups creation supported
  • ElasticSearch / Opensearch / Opensearch Serverless
    • VPC Support. Security Groups creation supported
  • Splunk
    • VPC Support. Security Groups creation supported
  • Custom Http Endpoint
  • DataDog
  • Coralogix
  • New Relic
  • Dynatrace
  • Honeycomb
  • Logic Monitor
  • MongoDB Cloud
  • Sumo Logic
• Data Transformation With Lambda
• Original Data Backup in S3
• Logging and Encryption
• Application Role to Direct Put Sources
• Turn on/off cloudwatch logs decompressing and data message extraction
• Permissions
  • IAM Roles
  • Opensearch / Opensearch Serverless Service Role
  • Associate Role to Redshift Cluster Iam Roles
  • Cross Account S3 Bucket Policy
  • Cross Account Elasticsearch / OpenSearch / Opensearch Serverless Service policy

How to Use

Sources

Kinesis Data Stream

To Enabled it: input_source = "kinesis".

module "firehose" {
  source                    = "fdmsantos/kinesis-firehose/aws"
  version                   = "x.x.x"
  name                      = "firehose-delivery-stream"
  input_source              =  "kinesis"
  kinesis_source_stream_arn = "<kinesis_stream_arn>"
  destination               = "s3" # or destination = "extended_s3"
  s3_bucket_arn             = "<bucket_arn>"
}

Kinesis Data Stream Encrypted

If Kinesis Data Stream is encrypted, it's necessary pass this info to module .

To Enabled It: input_source = "kinesis".

KMS Key: use kinesis_source_kms_arn variable to indicate the KMS Key to module add permissions to policy to decrypt the Kinesis Data Stream.

Direct Put

To Enabled it: input_source = "direct-put".

module "firehose" {
  source           = "fdmsantos/kinesis-firehose/aws"
  version          = "x.x.x"
  name             = "firehose-delivery-stream"
  input_source     = "direct-put"
  destination      = "s3" # or destination = "extended_s3"
  s3_bucket_arn    = "<bucket_arn>"
}

WAF

To Enabled it: input_source = "waf".

module "firehose" {
  source           = "fdmsantos/kinesis-firehose/aws"
  version          = "x.x.x"
  name             = "firehose-delivery-stream"
  input_source     = "waf"
  destination      = "s3" # or destination = "extended_s3"
  s3_bucket_arn    = "<bucket_arn>"
}

MSK

To Enabled it: input_source = "msk".

module "firehose" {
  source                 = "fdmsantos/kinesis-firehose/aws"
  version                = "x.x.x"
  name                   = "firehose-delivery-stream"
  input_source           = "msk"
  msk_source_cluster_arn = "<msk_cluster_arn>"
  msk_source_topic_name  = "test"
  destination            = "s3"
  s3_bucket_arn          = "<bucket_arn>"
}

Destinations

S3

To Enabled It: destination = "s3" or destination = "extended_s3"

Variables Prefix: s3_

To Enable Encryption: enable_s3_encryption = true

Note: For other destinations, the s3_ variables are used to configure the required intermediary bucket before delivery data to destination. Not Supported to Elasticsearch, Splunk and http destinations

module "firehose" {
  source                    = "fdmsantos/kinesis-firehose/aws"
  version                   = "x.x.x"
  name                      = "firehose-delivery-stream"
  destination               = "s3" # or destination = "extended_s3"
  s3_bucket_arn             = "<bucket_arn>"
}

Redshift

To Enabled It: destination = "redshift"

Variables Prefix: redshift_

module "firehose" {
  source                        = "fdmsantos/kinesis-firehose/aws"
  version                       = "x.x.x"
  name                          = "firehose-delivery-stream"
  destination                   = "redshift"
  s3_bucket_arn                 = "<bucket_arn>"
  redshift_cluster_identifier   = "<redshift_cluster_identifier>"
  redshift_cluster_endpoint     = "<redshift_cluster_endpoint>"
  redshift_database_name        = "<redshift_cluster_database>"
  redshift_username             = "<redshift_cluster_username>"
  redshift_password             = "<redshift_cluster_password>"
  redshift_table_name           = "<redshift_cluster_table>"
  redshift_copy_options         = "json 'auto ignorecase'"
}

Elasticsearch

To Enabled It: destination = "elasticsearch"

Variables Prefix: elasticsearch_

module "firehose" {
  source                   = "fdmsantos/kinesis-firehose/aws"
  version                  = "x.x.x"
  name                     = "firehose-delivery-stream"
  destination              = "elasticsearch"
  elasticsearch_domain_arn = "<elasticsearch_domain_arn>"
  elasticsearch_index_name = "<elasticsearch_index_name"
}

Opensearch

To Enabled It: destination = "opensearch"

Variables Prefix: opensearch_

module "firehose" {
  source                = "fdmsantos/kinesis-firehose/aws"
  version               = "x.x.x"
  name                  = "firehose-delivery-stream"
  destination           = "opensearch"
  opensearch_domain_arn = "<opensearch_domain_arn>"
  opensearch_index_name = "<opensearch_index_name"
}

Opensearch Serverless

To Enabled It: destination = "opensearchserverless"

Variables Prefix: opensearchserverless_

module "firehose" {
  source                                   = "fdmsantos/kinesis-firehose/aws"
  version                                  = "x.x.x"
  name                                     = "firehose-delivery-stream"
  destination                              = "opensearch"
  opensearchserverless_collection_endpoint = "<opensearchserverless_collection_endpoint>"
  opensearchserverless_collection_arn      = "<opensearchserverless_collection_arn>"
  opensearch_index_name                    = "<opensearchserverless_index_name"
}

Splunk

To Enabled It: destination = "splunk"

Variables Prefix: splunk_

module "firehose" {
  source                            = "fdmsantos/kinesis-firehose/aws"
  version                           = "x.x.x"
  name                              = "firehose-delivery-stream"
  destination                       = "splunk"
  splunk_hec_endpoint               = "<splunk_hec_endpoint>"
  splunk_hec_endpoint_type          = "<splunk_hec_endpoint_type>"
  splunk_hec_token                  = "<splunk_hec_token>"
  splunk_hec_acknowledgment_timeout = 450
  splunk_retry_duration             = 450
}

HTTP Endpoint

To Enabled It: destination = "http_endpoint"

Variables Prefix: http_endpoint_

To enable Request Configuration: http_endpoint_enable_request_configuration = true

Request Configuration Variables Prefix: http_endpoint_request_configuration_

module "firehose" {
  source                                                = "fdmsantos/kinesis-firehose/aws"
  version                                               = "x.x.x"
  name                                                  = "firehose-delivery-stream"
  destination                                           = "http_endpoint"
  buffer_interval                                       = 60
  http_endpoint_name                                    = "<http_endpoint_name>"
  http_endpoint_url                                     = "<http_endpoint_url>"
  http_endpoint_access_key                              = "<http_endpoint_access_key>"
  http_endpoint_retry_duration                          = 400
  http_endpoint_enable_request_configuration            = true
  http_endpoint_request_configuration_content_encoding  = "GZIP"
  http_endpoint_request_configuration_common_attributes = [
    {
      name  = "testname"
      value = "testvalue"
    },
    {
      name  = "testname2"
      value = "testvalue2"
    }
  ]
}

Datadog

To Enabled It: destination = "datadog"

Variables Prefix: http_endpoint_ and datadog_endpoint_type

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and datadog destinations

module "firehose" {
  source                   = "fdmsantos/kinesis-firehose/aws"
  version                  = "x.x.x"
  name                     = "firehose-delivery-stream"
  destination              = "datadog"
  datadog_endpoint_type    = "metrics_eu"
  http_endpoint_access_key = "<datadog_api_key>"
}

New Relic

To Enabled It: destination = "newrelic"

Variables Prefix: http_endpoint_ and newrelic_endpoint_type

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and newrelic destinations

module "firehose" {
  source                   = "fdmsantos/kinesis-firehose/aws"
  version                  = "x.x.x"
  name                     = "firehose-delivery-stream"
  destination              = "newrelic"
  newrelic_endpoint_type   = "metrics_eu"
  http_endpoint_access_key = "<newrelic_api_key>"
}

Coralogix

To Enabled It: destination = "coralogix"

Variables Prefix: http_endpoint_ and coralogix_

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and coralogix destinations

**Check Firehose-to-Coralogix to more details.

module "firehose" {
  source                       = "fdmsantos/kinesis-firehose/aws"
  version                      = "x.x.x"
  name                         = "firehose-delivery-stream"
  destination                  = "coralogix"
  coralogix_endpoint_location  = "ireland"
  http_endpoint_access_key     = "<coralogix_private_key>"
}

Dynatrace

To Enabled It: destination = "dynatrace"

Variables Prefix: http_endpoint_, dynatrace_endpoint_location and dynatrace_api_url

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and dynatrace destinations

module "firehose" {
  source                      = "fdmsantos/kinesis-firehose/aws"
  version                     = "x.x.x"
  name                        = "firehose-delivery-stream"
  destination                 = "dynatrace"
  dynatrace_endpoint_location = "eu"
  dynatrace_api_url           = "https://xyazb123456.live.dynatrace.com"
  http_endpoint_access_key    = "<dynatrace_api_token>"
}

Honeycomb

To Enabled It: destination = "honeycomb"

Variables Prefix: http_endpoint_, honeycomb_api_host (Default: https://api.honeycomb.io) and honeycomb_dataset_name.

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and honeycomb destinations

module "firehose" {
  source                   = "fdmsantos/kinesis-firehose/aws"
  version                  = "x.x.x"
  name                     = "firehose-delivery-stream"
  destination              = "honeycomb"
  honeycomb_api_host       = "https://api.honeycomb.io"
  honeycomb_dataset_name   = "<honeycomb_dataset_name>"
  http_endpoint_access_key = "<honeycomb_api_key>"
}

Logic Monitor

To Enabled It: destination = "logicmonitor"

Variables Prefix: http_endpoint_ and logicmonitor_account

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and logicmonitor destinations

module "firehose" {
  source                   = "fdmsantos/kinesis-firehose/aws"
  version                  = "x.x.x"
  name                     = "firehose-delivery-stream"
  destination              = "logicmonitor"
  logicmonitor_account     = "<logicmonitor_account>"
  http_endpoint_access_key = "<logicmonitor_api_key>"
}

MongoDB

To Enabled It: destination = "mongodb"

Variables Prefix: http_endpoint_ and mongodb_realm_webhook_url

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and mongodb destinations

module "firehose" {
  source                    = "fdmsantos/kinesis-firehose/aws"
  version                   = "x.x.x"
  name                      = "firehose-delivery-stream"
  destination               = "mongodb"
  mongodb_realm_webhook_url = "<mongodb_realm_webhook_url>"
  http_endpoint_access_key  = "<mongodb_api_key>"
}

SumoLogic

To Enabled It: destination = "sumologic"

Variables Prefix: http_endpoint_, sumologic_deployment_name and sumologic_data_type

Check HTTP Endpoint to more details and Destinations Mapping to see the difference between http_endpoint and Sumo Logic destinations

module "firehose" {
  source                    = "fdmsantos/kinesis-firehose/aws"
  version                   = "x.x.x"
  name                      = "firehose-delivery-stream"
  destination               = "sumologic"
  sumologic_deployment_name = "<sumologic_deployment_name>"
  sumologic_data_type       = "<sumologic_data_type>"
  http_endpoint_access_key  = "<sumologic_access_token>"
}

Server Side Encryption

Supported By: Only Direct Put source

To Enabled It: enable_sse = true

Variables Prefix: sse_

module "firehose" {
  source           = "fdmsantos/kinesis-firehose/aws"
  version          = "x.x.x"
  name             = "firehose-delivery-stream"
  destination      = "s3" # or destination = "extended_s3"
  s3_bucket_arn    = "<bucket_arn>"
  enable_sse       = true
  sse_kms_key_type = "CUSTOMER_MANAGED_CMK"
  sse_kms_key_arn  = aws_kms_key.this.arn
}

Data Transformation with Lambda

Supported By: All destinations and Sources

To Enabled It: enable_lambda_transform = true

Variables Prefix: transform_lambda_

module "firehose" {
  source                           = "fdmsantos/kinesis-firehose/aws"
  version                          = "x.x.x"
  name                             = "firehose-delivery-stream"
  input_source                     = "kinesis"
  kinesis_source_stream_arn        = "<kinesis_stream_arn>"
  destination                      = "s3" # or destination = "extended_s3"
  s3_bucket_arn                    = "<bucket_arn>"
  enable_lambda_transform          = true
  transform_lambda_arn             = "<lambda_arn>"
  transform_lambda_buffer_size     = 2 # Don't configure this parameter if you want use default value (1)
  transform_lambda_buffer_interval = 90 # Don't configure this parameter if you want use default value (60)
  transform_lambda_number_retries  = 4 # Don't configure this parameter if you want use default value (3)
}

Data Format Conversion

Supported By: Only S3 Destination

To Enabled It: enable_data_format_conversion = true

Variables Prefix: data_format_conversion_

module "firehose" {
  source                                 = "fdmsantos/kinesis-firehose/aws"
  version                                = "x.x.x"
  name                                   = "firehose-delivery-stream"
  input_source                           = "kinesis"
  kinesis_source_stream_arn              = "<kinesis_stream_arn>"
  destination                            = "s3" # or destination = "extended_s3"
  s3_bucket_arn                          = "<bucket_arn>"
  enable_data_format_conversion          = true
  data_format_conversion_glue_database   = "<glue_database_name>"
  data_format_conversion_glue_table_name = "<glue_table_name>"
  data_format_conversion_input_format    = "HIVE"
  data_format_conversion_output_format   = "ORC"
}

Dynamic Partition

Supported By: Only S3 Destination

To Enabled It: enable_dynamic_partitioning = true

Variables Prefix: dynamic_partitioning_

module "firehose" {
  source                                        = "fdmsantos/kinesis-firehose/aws"
  version                                       = "x.x.x"
  name                                          = "firehose-delivery-stream"
  input_source                                  = "kinesis"
  kinesis_source_stream_arn                     = "<kinesis_stream_arn>"
  destination                                   = "s3" # or destination = "extended_s3"
  s3_bucket_arn                                 = "<bucket_arn>"
  s3_prefix                                     = "prod/user_id=!{partitionKeyFromQuery:user_id}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
  enable_dynamic_partitioning                   = true
  dynamic_partitioning_retry_duration           = 350
  dynamic_partition_metadata_extractor_query    = "{user_id:.user_id}"
  dynamic_partition_enable_record_deaggregation = true
}

S3 Backup Data

Supported By: All Destinations

To Enabled It: enable_s3_backup = true. It's always enable to Elasticsearch, splunk and http destinations

To Enable Backup Encryption: s3_backup_enable_encryption = true

To Enable Backup Logging s3_backup_enable_log = true. Not supported to Elasticsearch, splunk and http destinations. It's possible add existing cloudwatch group or create new

Variables Prefix: s3_backup_

module "firehose" {
  source                        = "fdmsantos/kinesis-firehose/aws"
  version                       = "x.x.x"
  name                          = "${var.name_prefix}-delivery-stream"
  destination                   = "s3" # or destination = "extended_s3"
  s3_bucket_arn                 = aws_s3_bucket.s3.arn
  enable_s3_backup              = true
  s3_backup_bucket_arn          = aws_s3_bucket.s3.arn
  s3_backup_prefix              = "backup/"
  s3_backup_error_output_prefix = "error/"
  s3_backup_buffer_interval     = 100
  s3_backup_buffer_size         = 100
  s3_backup_compression         = "GZIP"
  s3_backup_use_existing_role   = false
  s3_backup_role_arn            = aws_iam_role.this.arn
  s3_backup_enable_encryption   = true
  s3_backup_kms_key_arn         = aws_kms_key.this.arn
  s3_backup_enable_log          = true
}

Destination Delivery Logging

Supported By: All Destinations

To Enabled It: enable_destination_log = true. It's enabled by default. It's possible add existing cloudwatch group or create new

Variables Prefix: destination_cw_log_

module "firehose" {
  source                      = "fdmsantos/kinesis-firehose/aws"
  version                     = "x.x.x"
  name                        = "firehose-delivery-stream"
  destination                 = "s3" # or destination = "extended_s3"
  s3_bucket_arn               = "<bucket_arn>"
  enable_destination_log      = true
  destination_log_group_name  = "<cw_log_group_arn>"
  destination_log_stream_name = "<cw_log_stream_name>"
}

VPC Support

It's possible use module only to create security groups.

Use variable create = false for this feature.

ElasticSearch / Opensearch / Opensearch Serverless

Supported By: ElasticSearch / Opensearch destination

To Enabled It: enable_vpc = true

To Create Opensearch IAM Service Linked Role: vpc_create_service_linked_role = true

If you want to have separate security groups for firehose and destination: vpc_security_group_same_as_destination = false

Examples

# Creates the Security Groups (For firehose and Destination)
module "firehose" {
  source                                               = "fdmsantos/kinesis-firehose/aws"
  version                                              = "x.x.x"
  name                                                 = "firehose-delivery-stream"
  destination                                          = "opensearch"
  opensearch_domain_arn                                = "<opensearch_domain_arn>"
  opensearch_index_name                                = "<opensearch_index_name>"
  enable_vpc                                           = true
  vpc_subnet_ids                                       = "<list(subnets_ids)>"
  vpc_create_security_group                            = true
  vpc_create_destination_security_group                = true
  elasticsearch_vpc_security_group_same_as_destination = false
}

# Use Existing Security Group
module "firehose" {
  source                          = "fdmsantos/kinesis-firehose/aws"
  version                         = "x.x.x"
  name                            = "firehose-delivery-stream"
  destination                     = "opensearch"
  opensearch_domain_arn           = "<opensearch_domain_arn>"
  opensearch_index_name           = "<opensearch_index_name>"
  enable_vpc                      = true
  vpc_subnet_ids                  = "<list(subnets_ids)>"
  vpc_security_group_firehose_ids = "<list(security_group_ids)>"
}

# Configure Existing Security Groups
module "firehose" {
  source                                            = "fdmsantos/kinesis-firehose/aws"
  version                                           = "x.x.x"
  name                                              = "firehose-delivery-stream"
  destination                                       = "opensearch"
  opensearch_domain_arn                             = "<opensearch_domain_arn>"
  opensearch_index_name                             = "<opensearch_index_name>"
  enable_vpc                                        = true
  vpc_subnet_ids                                    = "<list(subnets_ids)>"
  vpc_security_group_firehose_configure_existing    = true
  vpc_security_group_firehose_ids                   = "<list(security_group_ids)>"
  vpc_security_group_destination_configure_existing = true
  vpc_security_group_destination_ids                = "<list(security_group_ids)>"
}

Redshift / Splunk

Supported By: Redshift and Splunk destination

To Get Firehose CIDR Blocks to allow in destination security groups, use the following output: firehose_cidr_blocks

# Create the Security Group (For Destination)
module "firehose" {
  source                                = "fdmsantos/kinesis-firehose/aws"
  version                               = "x.x.x"
  name                                  = "firehose-delivery-stream"
  destination                           = "<redshift|splunk>"
  vpc_create_destination_security_group = true
}

# Configure Existing Security Groups
module "firehose" {
  source                                            = "fdmsantos/kinesis-firehose/aws"
  version                                           = "x.x.x"
  name                                              = "firehose-delivery-stream"
  destination                                       = "<redshift|splunk>"
  vpc_security_group_destination_configure_existing = true
  vpc_security_group_destination_ids                = "<list(security_group_ids)>"
}

Application Role

Supported By: Direct Put Source

To Create: create_application_role = true

To Create Policy: create_application_role_policy = true

Variables Prefix: application_role_

# Create Application Role to an application that runs in EC2 Instance
module "firehose" {
  source                             = "fdmsantos/kinesis-firehose/aws"
  version                            = "x.x.x"
  name                               = "firehose-delivery-stream"
  destination                        = "s3" # or destination = "extended_s3"
  create_application_role            = true
  create_application_role_policy     = true
  application_role_service_principal = "ec2.amazonaws.com"
}

# Configure existing Application Role to an application that runs in EC2 Instance with a policy with provided actions
module "firehose" {
  source                              = "fdmsantos/kinesis-firehose/aws"
  version                             = "x.x.x"
  name                                = "firehose-delivery-stream"
  destination                         = "s3" # or destination = "extended_s3"
  configure_existing_application_role = true
  application_role_name               = "application-role"
  create_application_role_policy      = true
  application_role_policy_actions     = [
    "firehose:PutRecord",
    "firehose:PutRecordBatch",
    "firehose:CreateDeliveryStream",
    "firehose:UpdateDestination"
  ]
}

Destinations Mapping

The destination variable configured in module is mapped to firehose valid destination.

Module Destination	Firehose Destination	Differences
s3 and extended_s3	extended_s3	There is no difference between s3 or extended_s3 destinations
redshift	redshift
splunk	splunk
opensearch	elasticsearch
opensearch	opensearch
opensearchserverless	opensearchserverless
http_endpoint	http_endpoint
datadog	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure datadog_endpoint_type variable
newrelic	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure newrelic_endpoint_type variable
coralogix	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure coralogix_endpoint_location variable
dynatrace	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure dynatrace_endpoint_location and dynatrace_api_url variable
honeycomb	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure honeycomb_dataset_name variable
logicmonitor	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure logicmonitor_account variable
mongodb	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure mongodb_realm_webhook_url variable
sumologic	http_endpoint	The difference regarding http_endpoint is the http_endpoint_url and http_endpoint_name variables aren't support, and it's necessary configure sumologic_deployment_name and sumologic_data_type variables

Examples

• Direct Put - Creates an encrypted Kinesis firehose stream with Direct Put as source and S3 as destination.
• Direct Put With Lambda - Creates a Kinesis firehose stream with Direct Put as source and S3 as destination with transformation lambda.
• Kinesis Data Stream Source - Creates a basic Kinesis Firehose stream with Kinesis data stream as source and s3 as destination.
• WAF Source - Creates a Kinesis Firehose Stream with AWS Web WAF as source and S3 as destination.
• MSK Source - Creates a Kinesis Firehose Stream with MSK Cluster as source and S3 as destination.
• S3 Destination Complete - Creates a Kinesis Firehose Stream with all features enabled.
• Redshift - Creates a Kinesis Firehose Stream with redshift as destination.
• Redshift In VPC - Creates a Kinesis Firehose Stream with redshift in VPC as destination.
• Public Opensearch - Creates a Kinesis Firehose Stream with public opensearch as destination.
• Public Opensearch Serverless - Creates a Kinesis Firehose Stream with public serverless opensearch as destination.
• Opensearch Serverless In Vpc - Creates a Kinesis Firehose Stream with serverless opensearch in VPC as destination.
• Public Splunk - Creates a Kinesis Firehose Stream with public splunk as destination.
• Splunk In VPC - Creates a Kinesis Firehose Stream with splunk in VPC as destination.
• Custom Http Endpoint - Creates a Kinesis Firehose Stream with custom http endpoint as destination.
• Datadog - Creates a Kinesis Firehose Stream with datadog europe metrics as destination.
• New Relic - Creates a Kinesis Firehose Stream with New Relic europe metrics as destination.
• Coralogix - Creates a Kinesis Firehose Stream with coralogix ireland as destination.
• Dynatrace - Creates a Kinesis Firehose Stream with dynatrace europe as destination.
• Honeycomb - Creates a Kinesis Firehose Stream with honeycomb as destination.
• LogicMonitor - Creates a Kinesis Firehose Stream with Logic Monitor as destination.
• MongoDB - Creates a Kinesis Firehose Stream with MongoDB as destination.
• SumoLogic - Creates a Kinesis Firehose Stream with Sumo Logic as destination.

Requirements

Name	Version
terraform	>= 0.13.1
aws	>= 5.33

Providers

Name	Version
aws	>= 5.33

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_group.log	resource
aws_cloudwatch_log_stream.backup	resource
aws_cloudwatch_log_stream.destination	resource
aws_iam_policy.application	resource
aws_iam_policy.cw	resource
aws_iam_policy.elasticsearch	resource
aws_iam_policy.glue	resource
aws_iam_policy.kinesis	resource
aws_iam_policy.lambda	resource
aws_iam_policy.msk	resource
aws_iam_policy.opensearch	resource
aws_iam_policy.opensearchserverless	resource
aws_iam_policy.s3	resource
aws_iam_policy.s3_kms	resource
aws_iam_policy.vpc	resource
aws_iam_role.application	resource
aws_iam_role.firehose	resource
aws_iam_role_policy_attachment.application	resource
aws_iam_role_policy_attachment.cw	resource
aws_iam_role_policy_attachment.elasticsearch	resource
aws_iam_role_policy_attachment.glue	resource
aws_iam_role_policy_attachment.kinesis	resource
aws_iam_role_policy_attachment.lambda	resource
aws_iam_role_policy_attachment.msk	resource
aws_iam_role_policy_attachment.opensearch	resource
aws_iam_role_policy_attachment.opensearchserverless	resource
aws_iam_role_policy_attachment.s3	resource
aws_iam_role_policy_attachment.s3_kms	resource
aws_iam_role_policy_attachment.vpc	resource
aws_iam_service_linked_role.opensearch	resource
aws_iam_service_linked_role.opensearchserverless	resource
aws_kinesis_firehose_delivery_stream.this	resource
aws_redshift_cluster_iam_roles.this	resource
aws_security_group.destination	resource
aws_security_group.firehose	resource
aws_security_group_rule.destination	resource
aws_security_group_rule.firehose	resource
aws_security_group_rule.firehose_egress_rule	resource
aws_caller_identity.current	data source
aws_iam_policy_document.application	data source
aws_iam_policy_document.application_assume_role	data source
aws_iam_policy_document.assume_role	data source
aws_iam_policy_document.cross_account_elasticsearch	data source
aws_iam_policy_document.cross_account_opensearch	data source
aws_iam_policy_document.cross_account_opensearchserverless	data source
aws_iam_policy_document.cross_account_s3	data source
aws_iam_policy_document.cw	data source
aws_iam_policy_document.elasticsearch	data source
aws_iam_policy_document.glue	data source
aws_iam_policy_document.kinesis	data source
aws_iam_policy_document.lambda	data source
aws_iam_policy_document.msk	data source
aws_iam_policy_document.opensearch	data source
aws_iam_policy_document.opensearchserverless	data source
aws_iam_policy_document.s3	data source
aws_iam_policy_document.s3_kms	data source
aws_iam_policy_document.vpc	data source
aws_region.current	data source
aws_subnet.subnet	data source

Inputs

Name	Description	Type	Default	Required
append_delimiter_to_record	To configure your delivery stream to add a new line delimiter between records in objects that are delivered to Amazon S3.	bool	false	no
application_role_description	Description of IAM Application role to use for Kinesis Firehose Stream Source	string	null	no
application_role_force_detach_policies	Specifies to force detaching any policies the IAM Application role has before destroying it	bool	true	no
application_role_name	Name of IAM Application role to use for Kinesis Firehose Stream Source	string	null	no
application_role_path	Path of IAM Application role to use for Kinesis Firehose Stream Source	string	null	no
application_role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the IAM Application role used by Kinesis Firehose Stream Source	string	null	no
application_role_policy_actions	List of Actions to Application Role Policy	list(string)	[ "firehose:PutRecord", "firehose:PutRecordBatch" ]	no
application_role_service_principal	AWS Service Principal to assume application role	string	null	no
application_role_tags	A map of tags to assign to IAM Application role	map(string)	{}	no
associate_role_to_redshift_cluster	Set it to false if don't want the module associate the role to redshift cluster	bool	true	no
buffering_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination	number	300	no
buffering_size	Buffer incoming data to the specified size, in MBs, before delivering it to the destination.	number	5	no
configure_existing_application_role	Set it to True if want use existing application role to add the firehose Policy	bool	false	no
coralogix_endpoint_location	Endpoint Location to coralogix destination	string	"ireland"	no
coralogix_parameter_application_name	By default, your delivery stream arn will be used as applicationName	string	null	no
coralogix_parameter_subsystem_name	By default, your delivery stream name will be used as subsystemName	string	null	no
coralogix_parameter_use_dynamic_values	To use dynamic values for applicationName and subsystemName	bool	false	no
create	Controls if kinesis firehose should be created (it affects almost all resources)	bool	true	no
create_application_role	Set it to true to create role to be used by the source	bool	false	no
create_application_role_policy	Set it to true to create policy to the role used by the source	bool	false	no
create_destination_cw_log_group	Enables or disables the cloudwatch log group creation to destination	bool	true	no
create_role	Controls whether IAM role for Kinesis Firehose Stream should be created	bool	true	no
cw_log_retention_in_days	Specifies the number of days you want to retain log events in the specified log group. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653.	number	null	no
cw_tags	A map of tags to assign to the resource.	map(string)	{}	no
data_format_conversion_block_size	The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The Value is in Bytes.	number	268435456	no
data_format_conversion_glue_catalog_id	The ID of the AWS Glue Data Catalog. If you don't supply this, the AWS account ID is used by default.	string	null	no
data_format_conversion_glue_database	Name of the AWS Glue database that contains the schema for the output data.	string	null	no
data_format_conversion_glue_region	If you don't specify an AWS Region, the default is the current region.	string	null	no
data_format_conversion_glue_role_arn	The role that Kinesis Data Firehose can use to access AWS Glue. This role must be in the same account you use for Kinesis Data Firehose. Cross-account roles aren't allowed.	string	null	no
data_format_conversion_glue_table_name	Specifies the AWS Glue table that contains the column information that constitutes your data schema	string	null	no
data_format_conversion_glue_use_existing_role	Indicates if want use the kinesis firehose role to glue access.	bool	true	no
data_format_conversion_glue_version_id	Specifies the table version for the output data schema.	string	"LATEST"	no
data_format_conversion_hive_timestamps	A list of how you want Kinesis Data Firehose to parse the date and time stamps that may be present in your input data JSON. To specify these format strings, follow the pattern syntax of JodaTime's DateTimeFormat format strings.	list(string)	[]	no
data_format_conversion_input_format	Specifies which deserializer to use. You can choose either the Apache Hive JSON SerDe or the OpenX JSON SerDe	string	"OpenX"	no
data_format_conversion_openx_case_insensitive	When set to true, Kinesis Data Firehose converts JSON keys to lowercase before deserializing them.	bool	true	no
data_format_conversion_openx_column_to_json_key_mappings	A map of column names to JSON keys that aren't identical to the column names. This is useful when the JSON contains keys that are Hive keywords.	map(string)	null	no
data_format_conversion_openx_convert_dots_to_underscores	Specifies that the names of the keys include dots and that you want Kinesis Data Firehose to replace them with underscores. This is useful because Apache Hive does not allow dots in column names.	bool	false	no
data_format_conversion_orc_bloom_filter_columns	A list of column names for which you want Kinesis Data Firehose to create bloom filters.	list(string)	[]	no
data_format_conversion_orc_bloom_filter_false_positive_probability	The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter.	number	0.05	no
data_format_conversion_orc_compression	The compression code to use over data blocks.	string	"SNAPPY"	no
data_format_conversion_orc_dict_key_threshold	A float that represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.	number	0	no
data_format_conversion_orc_enable_padding	Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying.	bool	false	no
data_format_conversion_orc_format_version	The version of the file to write.	string	"V0_12"	no
data_format_conversion_orc_padding_tolerance	A float between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.	number	0.05	no
data_format_conversion_orc_row_index_stripe	The number of rows between index entries.	number	10000	no
data_format_conversion_orc_stripe_size	he number of bytes in each strip.	number	67108864	no
data_format_conversion_output_format	Specifies which serializer to use. You can choose either the ORC SerDe or the Parquet SerDe	string	"PARQUET"	no
data_format_conversion_parquet_compression	The compression code to use over data blocks.	string	"SNAPPY"	no
data_format_conversion_parquet_dict_compression	Indicates whether to enable dictionary compression.	bool	false	no
data_format_conversion_parquet_max_padding	The maximum amount of padding to apply. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The value is in bytes	number	0	no
data_format_conversion_parquet_page_size	Column chunks are divided into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). The value is in bytes	number	1048576	no
data_format_conversion_parquet_writer_version	Indicates the version of row format to output.	string	"V1"	no
datadog_endpoint_type	Endpoint type to datadog destination	string	"logs_eu"	no
destination	This is the destination to where the data is delivered	string	n/a	yes
destination_cross_account	Indicates if destination is in a different account. Only supported to Elasticsearch and OpenSearch	bool	false	no
destination_log_group_name	The CloudWatch group name for destination logs	string	null	no
destination_log_stream_name	The CloudWatch log stream name for destination logs	string	null	no
dynamic_partition_append_delimiter_to_record	DEPRECATED!! Use var append_delimiter_to_record instead!! Use To configure your delivery stream to add a new line delimiter between records in objects that are delivered to Amazon S3.	bool	false	no
dynamic_partition_enable_record_deaggregation	Data deaggregation is the process of parsing through the records in a delivery stream and separating the records based either on valid JSON or on the specified delimiter	bool	false	no
dynamic_partition_metadata_extractor_query	Dynamic Partition JQ query.	string	null	no
dynamic_partition_record_deaggregation_delimiter	Specifies the delimiter to be used for parsing through the records in the delivery stream and deaggregating them	string	null	no
dynamic_partition_record_deaggregation_type	Data deaggregation is the process of parsing through the records in a delivery stream and separating the records based either on valid JSON or on the specified delimiter	string	"JSON"	no
dynamic_partitioning_retry_duration	Total amount of seconds Firehose spends on retries	number	300	no
dynatrace_api_url	API URL to Dynatrace destination	string	null	no
dynatrace_endpoint_location	Endpoint Location to Dynatrace destination	string	"eu"	no
elasticsearch_domain_arn	The ARN of the Amazon ES domain. The pattern needs to be arn:.*	string	null	no
elasticsearch_index_name	The Elasticsearch index name	string	null	no
elasticsearch_index_rotation_period	The Elasticsearch index rotation period. Index rotation appends a timestamp to the IndexName to facilitate expiration of old data	string	"OneDay"	no
elasticsearch_retry_duration	The length of time during which Firehose retries delivery after a failure, starting from the initial request and including the first attempt	string	300	no
elasticsearch_type_name	The Elasticsearch type name with maximum length of 100 characters	string	null	no
enable_cloudwatch_logs_data_message_extraction	Cloudwatch Logs data message extraction	bool	false	no
enable_cloudwatch_logs_decompression	Enables or disables Cloudwatch Logs decompression	bool	false	no
enable_data_format_conversion	Set it to true if you want to disable format conversion.	bool	false	no
enable_destination_log	The CloudWatch Logging Options for the delivery stream	bool	true	no
enable_dynamic_partitioning	Enables or disables dynamic partitioning	bool	false	no
enable_lambda_transform	Set it to true to enable data transformation with lambda	bool	false	no
enable_s3_backup	The Amazon S3 backup mode	bool	false	no
enable_s3_encryption	Indicates if want use encryption in S3 bucket.	bool	false	no
enable_sse	Whether to enable encryption at rest. Only makes sense when source is Direct Put	bool	false	no
enable_vpc	Indicates if destination is configured in VPC. Supports Elasticsearch and Opensearch destinations.	bool	false	no
firehose_role	IAM role ARN attached to the Kinesis Firehose Stream.	string	null	no
honeycomb_api_host	If you use a Secure Tenancy or other proxy, put its schema://host[:port] here	string	"https://api.honeycomb.io"	no
honeycomb_dataset_name	Your Honeycomb dataset name to Honeycomb destination	string	null	no
http_endpoint_access_key	The access key required for Kinesis Firehose to authenticate with the HTTP endpoint selected as the destination	string	null	no
http_endpoint_enable_request_configuration	The request configuration	bool	false	no
http_endpoint_name	The HTTP endpoint name	string	null	no
http_endpoint_request_configuration_common_attributes	Describes the metadata sent to the HTTP endpoint destination. The variable is list. Each element is map with two keys , name and value, that corresponds to common attribute name and value	list(map(string))	[]	no
http_endpoint_request_configuration_content_encoding	Kinesis Data Firehose uses the content encoding to compress the body of a request before sending the request to the destination	string	"GZIP"	no
http_endpoint_retry_duration	Total amount of seconds Firehose spends on retries. This duration starts after the initial attempt fails, It does not include the time periods during which Firehose waits for acknowledgment from the specified destination after each attempt	number	300	no
http_endpoint_url	The HTTP endpoint URL to which Kinesis Firehose sends your data	string	null	no
input_source	This is the kinesis firehose source	string	"direct-put"	no
kinesis_source_is_encrypted	Indicates if Kinesis data stream source is encrypted	bool	false	no
kinesis_source_kms_arn	Kinesis Source KMS Key to add Firehose role to decrypt the records.	string	null	no
kinesis_source_role_arn	DEPRECATED!! Use variable instead source_role_arn! The ARN of the role that provides access to the source Kinesis stream	string	null	no
kinesis_source_stream_arn	The kinesis stream used as the source of the firehose delivery stream	string	null	no
kinesis_source_use_existing_role	DEPRECATED!! Use variable source_use_existing_role instead! Indicates if want use the kinesis firehose role to kinesis data stream access.	bool	true	no
logicmonitor_account	Account to use in Logic Monitor destination	string	null	no
mongodb_realm_webhook_url	Realm Webhook URL to use in MongoDB destination	string	null	no
msk_source_cluster_arn	The ARN of the Amazon MSK cluster.	string	null	no
msk_source_connectivity_type	The type of connectivity used to access the Amazon MSK cluster. Valid values: PUBLIC, PRIVATE.	string	"PUBLIC"	no
msk_source_topic_name	The topic name within the Amazon MSK cluster.	string	null	no
name	A name to identify the stream. This is unique to the AWS account and region the Stream is created in	string	n/a	yes
newrelic_endpoint_type	Endpoint type to New Relic destination	string	"logs_eu"	no
opensearch_document_id_options	The method for setting up document ID.	string	"FIREHOSE_DEFAULT"	no
opensearch_domain_arn	The ARN of the Amazon Opensearch domain. The pattern needs to be arn:.*. Conflicts with cluster_endpoint.	string	null	no
opensearch_index_name	The Opensearch (And OpenSearch Serverless) index name.	string	null	no
opensearch_index_rotation_period	The Opensearch index rotation period. Index rotation appends a timestamp to the IndexName to facilitate expiration of old data	string	"OneDay"	no
opensearch_retry_duration	After an initial failure to deliver to Amazon OpenSearch, the total amount of time, in seconds between 0 to 7200, during which Firehose re-attempts delivery (including the first attempt). After this time has elapsed, the failed documents are written to Amazon S3. The default value is 300s. There will be no retry if the value is 0.	string	300	no
opensearch_type_name	The opensearch type name with maximum length of 100 characters. Types are deprecated in OpenSearch_1.1. TypeName must be empty.	string	null	no
opensearch_vpc_create_service_linked_role	Set it to True if want create Opensearch Service Linked Role to Access VPC.	bool	false	no
opensearchserverless_collection_arn	The ARN of the Amazon Opensearch Serverless Collection. The pattern needs to be arn:.*.	string	null	no
opensearchserverless_collection_endpoint	The endpoint to use when communicating with the collection in the Serverless offering for Amazon OpenSearch Service.	string	null	no
policy_path	Path of policies to that should be added to IAM role for Kinesis Firehose Stream	string	null	no
redshift_cluster_endpoint	The redshift endpoint	string	null	no
redshift_cluster_identifier	Redshift Cluster identifier. Necessary to associate the iam role to cluster	string	null	no
redshift_copy_options	Copy options for copying the data from the s3 intermediate bucket into redshift, for example to change the default delimiter	string	null	no
redshift_data_table_columns	The data table columns that will be targeted by the copy command	string	null	no
redshift_database_name	The redshift database name	string	null	no
redshift_password	The password for the redshift username above	string	null	no
redshift_retry_duration	The length of time during which Firehose retries delivery after a failure, starting from the initial request and including the first attempt	string	3600	no
redshift_table_name	The name of the table in the redshift cluster that the s3 bucket will copy to	string	null	no
redshift_username	The username that the firehose delivery stream will assume. It is strongly recommended that the username and password provided is used exclusively for Amazon Kinesis Firehose purposes, and that the permissions for the account are restricted for Amazon Redshift INSERT permissions	string	null	no
role_description	Description of IAM role to use for Kinesis Firehose Stream	string	null	no
role_force_detach_policies	Specifies to force detaching any policies the IAM role has before destroying it	bool	true	no
role_name	Name of IAM role to use for Kinesis Firehose Stream	string	null	no
role_path	Path of IAM role to use for Kinesis Firehose Stream	string	null	no
role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the IAM role used by Kinesis Firehose Stream	string	null	no
role_tags	A map of tags to assign to IAM role	map(string)	{}	no
s3_backup_bucket_arn	The ARN of the S3 backup bucket	string	null	no
s3_backup_buffering_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination.	number	300	no
s3_backup_buffering_size	Buffer incoming data to the specified size, in MBs, before delivering it to the destination.	number	5	no
s3_backup_compression	The compression format	string	"UNCOMPRESSED"	no
s3_backup_create_cw_log_group	Enables or disables the cloudwatch log group creation	bool	true	no
s3_backup_enable_encryption	Indicates if want enable KMS Encryption in S3 Backup Bucket.	bool	false	no
s3_backup_enable_log	Enables or disables the logging	bool	true	no
s3_backup_error_output_prefix	Prefix added to failed records before writing them to S3	string	null	no
s3_backup_kms_key_arn	Specifies the KMS key ARN the stream will use to encrypt data. If not set, no encryption will be used.	string	null	no
s3_backup_log_group_name	he CloudWatch group name for logging	string	null	no
s3_backup_log_stream_name	The CloudWatch log stream name for logging	string	null	no
s3_backup_mode	Defines how documents should be delivered to Amazon S3. Used to elasticsearch, opensearch, splunk, http configurations. For S3 and Redshift use enable_s3_backup	string	"FailedOnly"	no
s3_backup_prefix	The YYYY/MM/DD/HH time format prefix is automatically used for delivered S3 files. You can specify an extra prefix to be added in front of the time format prefix. Note that if the prefix ends with a slash, it appears as a folder in the S3 bucket	string	null	no
s3_backup_role_arn	The role that Kinesis Data Firehose can use to access S3 Backup.	string	null	no
s3_backup_use_existing_role	Indicates if want use the kinesis firehose role to s3 backup bucket access.	bool	true	no
s3_bucket_arn	The ARN of the S3 destination bucket	string	null	no
s3_compression_format	The compression format	string	"UNCOMPRESSED"	no
s3_configuration_buffering_interval	Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination.	number	300	no
s3_configuration_buffering_size	Buffer incoming data to the specified size, in MBs, before delivering it to the destination. The default value is 5. We recommend setting SizeInMBs to a value greater than the amount of data you typically ingest into the delivery stream in 10 seconds. For example, if you typically ingest data at 1 MB/sec set SizeInMBs to be 10 MB or higher.	number	5	no
s3_cross_account	Indicates if S3 bucket destination is in a different account	bool	false	no
s3_error_output_prefix	Prefix added to failed records before writing them to S3. This prefix appears immediately following the bucket name.	string	null	no
s3_kms_key_arn	Specifies the KMS key ARN the stream will use to encrypt data. If not set, no encryption will be used	string	null	no
s3_own_bucket	Indicates if you own the bucket. If not, will be configure permissions to grants the bucket owner full access to the objects delivered by Kinesis Data Firehose	bool	true	no
s3_prefix	The YYYY/MM/DD/HH time format prefix is automatically used for delivered S3 files. You can specify an extra prefix to be added in front of the time format prefix. Note that if the prefix ends with a slash, it appears as a folder in the S3 bucket	string	null	no
source_role_arn	The ARN of the role that provides access to the source. Only Supported on Kinesis and MSK Sources	string	null	no
source_use_existing_role	Indicates if want use the kinesis firehose role for sources access. Only Supported on Kinesis and MSK Sources	bool	true	no
splunk_hec_acknowledgment_timeout	The amount of time, that Kinesis Firehose waits to receive an acknowledgment from Splunk after it sends it data	number	600	no
splunk_hec_endpoint	The HTTP Event Collector (HEC) endpoint to which Kinesis Firehose sends your data	string	null	no
splunk_hec_endpoint_type	The HEC endpoint type	string	"Raw"	no
splunk_hec_token	The GUID that you obtain from your Splunk cluster when you create a new HEC endpoint	string	null	no
splunk_retry_duration	After an initial failure to deliver to Splunk, the total amount of time, in seconds between 0 to 7200, during which Firehose re-attempts delivery (including the first attempt)	number	300	no
sse_kms_key_arn	Amazon Resource Name (ARN) of the encryption key	string	null	no
sse_kms_key_type	Type of encryption key.	string	"AWS_OWNED_CMK"	no
sumologic_data_type	Data Type to use in Sumo Logic destination	string	"log"	no
sumologic_deployment_name	Deployment Name to use in Sumo Logic destination	string	null	no
tags	A map of tags to assign to resources.	map(string)	{}	no
transform_lambda_arn	Lambda ARN to Transform source records	string	null	no
transform_lambda_buffer_interval	The period of time during which Kinesis Data Firehose buffers incoming data before invoking the AWS Lambda function. The AWS Lambda function is invoked once the value of the buffer size or the buffer interval is reached.	number	null	no
transform_lambda_buffer_size	The AWS Lambda function has a 6 MB invocation payload quota. Your data can expand in size after it's processed by the AWS Lambda function. A smaller buffer size allows for more room should the data expand after processing.	number	null	no
transform_lambda_number_retries	Number of retries for AWS Transformation lambda	number	null	no
transform_lambda_role_arn	The ARN of the role to execute the transform lambda. If null use the Firehose Stream role	string	null	no
vpc_create_destination_security_group	Indicates if want create destination security group to associate to firehose destinations	bool	false	no
vpc_create_security_group	Indicates if want create security group to associate to kinesis firehose	bool	false	no
vpc_role_arn	The ARN of the IAM role to be assumed by Firehose for calling the Amazon EC2 configuration API and for creating network interfaces. Supports Elasticsearch and Opensearch destinations.	string	null	no
vpc_security_group_destination_configure_existing	Indicates if want configure an existing destination security group with the necessary rules	bool	false	no
vpc_security_group_destination_ids	A list of security group IDs associated to destinations to allow firehose traffic	list(string)	null	no
vpc_security_group_destination_vpc_id	VPC ID to create the destination security group. Only supported to Redshift and splunk destinations	string	null	no
vpc_security_group_firehose_configure_existing	Indicates if want configure an existing firehose security group with the necessary rules	bool	false	no
vpc_security_group_firehose_ids	A list of security group IDs to associate with Kinesis Firehose.	list(string)	null	no
vpc_security_group_same_as_destination	Indicates if the firehose security group is the same as destination.	bool	true	no
vpc_security_group_tags	A map of tags to assign to security group	map(string)	{}	no
vpc_subnet_ids	A list of subnet IDs to associate with Kinesis Firehose. Supports Elasticsearch and Opensearch destinations.	list(string)	null	no
vpc_use_existing_role	Indicates if want use the kinesis firehose role to VPC access. Supports Elasticsearch and Opensearch destinations.	bool	true	no

Outputs

Name	Description
application_role_arn	The ARN of the IAM role created for Kinesis Firehose Stream Source
application_role_name	The Name of the IAM role created for Kinesis Firehose Stream Source Source
application_role_policy_arn	The ARN of the IAM policy created for Kinesis Firehose Stream Source
application_role_policy_name	The Name of the IAM policy created for Kinesis Firehose Stream Source Source
destination_security_group_id	Security Group ID associated to destination
destination_security_group_name	Security Group Name associated to destination
destination_security_group_rule_ids	Security Group Rules ID created in Destination Security group
elasticsearch_cross_account_service_policy	Elasticsearch Service policy when the opensearch domain belongs to another account
firehose_cidr_blocks	Firehose stream cidr blocks to unblock on destination security group
firehose_security_group_id	Security Group ID associated to Firehose Stream. Only Supported for elasticsearch destination
firehose_security_group_name	Security Group Name associated to Firehose Stream. Only Supported for elasticsearch destination
firehose_security_group_rule_ids	Security Group Rules ID created in Firehose Stream Security group. Only Supported for elasticsearch destination
kinesis_firehose_arn	The ARN of the Kinesis Firehose Stream
kinesis_firehose_cloudwatch_log_backup_stream_arn	The ARN of the created Cloudwatch Log Group Stream to backup
kinesis_firehose_cloudwatch_log_backup_stream_name	The name of the created Cloudwatch Log Group Stream to backup
kinesis_firehose_cloudwatch_log_delivery_stream_arn	The ARN of the created Cloudwatch Log Group Stream to delivery
kinesis_firehose_cloudwatch_log_delivery_stream_name	The name of the created Cloudwatch Log Group Stream to delivery
kinesis_firehose_cloudwatch_log_group_arn	The ARN of the created Cloudwatch Log Group
kinesis_firehose_cloudwatch_log_group_name	The name of the created Cloudwatch Log Group
kinesis_firehose_destination_id	The Destination id of the Kinesis Firehose Stream
kinesis_firehose_name	The name of the Kinesis Firehose Stream
kinesis_firehose_role_arn	The ARN of the IAM role created for Kinesis Firehose Stream
kinesis_firehose_version_id	The Version id of the Kinesis Firehose Stream
opensearch_cross_account_service_policy	Opensearch Service policy when the opensearch domain belongs to another account
opensearch_iam_service_linked_role_arn	The ARN of the Opensearch IAM Service linked role
opensearchserverless_cross_account_service_policy	Opensearch Serverless Service policy when the opensearch domain belongs to another account
opensearchserverless_iam_service_linked_role_arn	The ARN of the Opensearch Serverless IAM Service linked role
s3_cross_account_bucket_policy	Bucket Policy to S3 Bucket Destination when the bucket belongs to another account

Upgrade

• Version 1.x to 2.x Upgrade Guide here
• Version 2.x to 3.x Upgrade Guide here

Deprecations

Version 3.1.0

• Variable kinesis_source_role_arn is deprecated. Use source_role_arn instead.
• Variable kinesis_source_use_existing_role is deprecated. Use source_use_existing_role instead.

Version 3.3.0

• Variable dynamic_partition_append_delimiter_to_record is deprecated. Use append_delimiter_to_record instead.

License

Apache 2 Licensed. See LICENSE for full details.