-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a new section on how to collect AWS Network Firewall using Firehose #3885
base: main
Are you sure you want to change the base?
Conversation
A documentation preview will be available soon. Request a new doc build by commenting
If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here. |
== Step 2: Select a resource | ||
|
||
You can either use an existing AWS Network Firewall, or create a new one. | ||
|
||
To create a new AWS Network Firewall, follow these steps: | ||
|
||
. Open the VPC service in the AWS console. | ||
+ | ||
The best option to create a VPC for a quick test is to use the wizard in the AWS console. | ||
|
||
. Create a VPC and other networking resources. | ||
+ | ||
Leave the default settings and choose a name for your VPC resources. | ||
+ | ||
[role="screenshot"] | ||
image::firehose-firewall-vpc-resources.png[Firewall VPC resources] | ||
|
||
. Create an AWS Network Firewall | ||
|
||
. Set up the firewall policy by creating a rule group. | ||
|
||
. Deploy an EC2 to generate network traffic. | ||
+ | ||
Launch an EC2, select the VPC you have just created, and enable *Auto-assign public IP*. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@alaudazzi, as we discussed offline, in the first version of this tutorial, we can assume the reader already has a Network Firewall producing logs they want to forward to an Elastic stack using Firehose. Creating a VPC for testing a Network Firewall is nontrivial and would take over the tutorial.
I am working on a dedicated zmoog/public-notes#87 to set up a testing Network Firewall using Terraform, so we can provide a one-step recipe to create a VPC with a Network Firewall for testing.
. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`. | ||
|
||
. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later. | ||
|
||
NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Creating a Firehose stream to forward Network Firewall is similar to the CloudTrail step.
. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Unless you are streaming data from Kinesis Data Streams, set source to `Direct PUT` and destination to `Elastic`. | |
. Provide a meaningful *Firehose stream name* that will allow you to identify this delivery stream later. Your Firehose name must start with the prefix `aws-waf-logs-` or it will not show up later. | |
NOTE: For advanced use cases, source records can be transformed by invoking a custom Lambda function. When using Elastic integrations, this should not be required. | |
For more information on how to set up a Amazon Data Firehose delivery stream to send data to Elastic Cloud, you can also check the <<monitor-aws-firehose,setup guide>>. | |
. Click *Create Firehose stream* and choose the source and destination of your Firehose stream. Set source to `Direct PUT` and destination to `Elastic`. | |
. Collect {es} endpoint and API key from your deployment on Elastic Cloud. | |
- Elastic endpoint URL: Enter the Elasticsearch endpoint URL of your Elasticsearch cluster. To find the Elasticsearch endpoint, go to the Elastic Cloud console and select *Connection details*. | |
- API key: Enter the encoded Elastic API key. To create an API key, go to the Elastic Cloud console, select *Connection details* and click *Create and manage API keys*. If you are using an API key with *Restrict privileges*, make sure to review the Indices privileges to provide at least "auto_configure" & "write" permissions for the indices you will be using with this delivery stream. | |
. Set up the delivery stream by specifying the following data: | |
+ | |
- Elastic endpoint URL | |
- API key | |
- Content encoding: gzip | |
- Retry duration: 60 (default) | |
- Parameters: | |
- es_datastream_name: `logs-aws.firewall_logs-default` | |
- Backup settings: failed data only to s3 bucket |
Here are a draft for the step four: [discrete]
[[firehose-cloudtrail-step-four]]
== Step 4: Enable logging
The AWS Network Firewall logs has logging support built in. It supports sending logs to Amazon S3, Amazon CloudWatch, and Amazon Kinesis Data Firehose.
To enable logging to Amazon Data Firehose:
- In the AWS console, navigate to the AWS Network Firewall service.
- Select the firewall you want to enable logging for.
- In the *Logging* section, click *Edit*.
- Select the *Send logs to* option and choose *Kinesis Data Firehose*.
- Select the Firehose stream you created in the previous step.
- Click *Save*. A note on the "Select the Send logs to option and choose Kinesis Data Firehose" step. Today, the AWS console still uses the old name "Kinesis Data Firehose" instead of the updated "Amazon Data Firehose". So this is probably one of those cases where we need to abstract the guide away from the UI details. |
. Set up logging. | ||
+ | ||
Open the *Logging* section to edit your firewall settings. If you want to quickly check your Network Firewall logs before setting up Firehose, you can enable logging on CloudWatch, and then inspect the log events: | ||
+ | ||
[role="screenshot"] | ||
image::firehose-firewall-logging.png[Firewall setup logging] | ||
|
||
. Visit CloudWatch and open your log group. If everything is working correctly, you will get the list of log events: | ||
+ | ||
[role="screenshot"] | ||
image::firehose-cloudwatch-log-events.png[CloudWatch Log events] | ||
|
||
[discrete] | ||
[[firehose-firewall-step-three]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to quickly check your Network Firewall logs before setting up Firehose, you can enable logging on CloudWatch, and then inspect the log events:
After rereading this part, I realized it isn't worth enabling logging on to CloudWatch. Enabling logging on CloudWatch brings value if we set up our test Network Firewall, and we want to double-check that our setup is sound and that it's actually logging data.
Since we assume the reader already has a working Network Firewall, I suggest setting up the logging to Firehose in step four.
Thank you for your comments @zmoog. I'll be on PTO for the next two weeks, in case you need support you can reach out to @dedemorton (thank you DeDe!) |
- Drop how to create a network firewall; it's too complex to include in guide. - Expand the guide with the missing content.
This pull request is now in conflict. Could you fix it @alaudazzi? 🙏
|
[role="screenshot"] | ||
image::firehose-networkfirewall-firewall.png[AWS Network Firewall] | ||
|
||
You can either use an existing AWS Network Firewall, or create a new one for testing purposes. | ||
|
||
Creating a Network Firewall is not trivial and is beyond the scope of this guide. For more information, see the AWS documentation on the https://docs.aws.amazon.com/network-firewall/latest/developerguide/getting-started.html[Getting started with AWS Network Firewall] guide. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dedemorton, explaining how to set up a Network Firewall for testing would be 2x the size of this guide.
So, I guess we have (at least) two options:
- say, "it's beyond the scope of this guide."
- link to an external source
I created a Terraform module at https://github.com/zmoog/integrations-cookbook/pull/1/files to test the Network Firewall for this guide. The module would allow users to set up the entire Network Firewall in one step.
However, (1) the module is in draft on (2) my personal repo. For the draft, I can finish it up quickly. But I can't link to my personal repo from the official Elastic docs.
What are our options here? Can we deliver the Terraform module through an official Elastic resource? Any suggestion is welcome!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question! TBH it's been quite awhile since I've provided users with links to special resources. Does it need to be a GitHub repo? I know in the past we've used download.elastic.co to provide sample data for guides, but this seems different. @bmorelli25 Do you have any ideas for the best way to deliver the Terraform module that Maurizio created for testing the network firewall?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @zmoog. Yes, we have the https://github.com/elastic/observability-examples repo for exactly this purpose.
[[firehose-firewall-step-two]] | ||
== Step 2: Select a resource | ||
|
||
[role="screenshot"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These will look better without our screenshot CSS since they're on a white background
[role="screenshot"] |
[[firehose-firewall-step-three]] | ||
== Step 3: Create a stream in Amazon Data Firehose | ||
|
||
[role="screenshot"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[role="screenshot"] |
[[firehose-firewall-step-four]] | ||
== Step 4: Enable logging | ||
|
||
[role="screenshot"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[role="screenshot"] |
|
||
- *Visualize your logs with Discover* | ||
+ | ||
image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And we should add it here because this one is a screenshot
image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover] | |
[role="screenshot"] | |
image::firehose-networkfirewall-discover.png[Visualize Network Firewall logs with Discover] |
Navigate to {kib} and choose among the following monitoring options: | ||
|
||
- *Visualize your logs with Discover* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We try to avoid lists of one. How about this?
Navigate to {kib} and choose among the following monitoring options: | |
- *Visualize your logs with Discover* | |
Navigate to {kib} and choose *Visualize your logs with Discover*. |
With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream. | ||
|
||
image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move image to the top like in the other sections.
With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream. | |
image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs] | |
image::firehose-networkfirewall-data-stream.png[Firehose monitor Network Firewall logs] | |
With the new logging settings in place, the Network Firewall starts sending log events to the Firehose stream. |
This PR:
Doc preview
Closes #3881