Table of Contents
- 1. Introduction
- 2. Changes
- 3. Adjustments to make when switching to S3 method with existing Lambda
- 4. Things to keep in mind
- 5. Conclusion
1. Introduction
On May 11th, 2022 (JST), WafCharm was updated to accept WAF logs directly published to an S3 bucket for our Reporting & Notification feature.
Previously, WAF logs had to be sent through Kinesis Data Firehose to enable Reporting & Notification feature. With this update, you can publish WAF logs directly to S3 to use the same function.
In this post, we will talk about changes and things to keep in mind when switching to the “S3 method” if you already have the Reporting & Notification feature enabled with Kinesis Data Firehose.
2. Changes
2-1 Architecture
First, please take a look at the architectures of each method.
Kinesis Data Firehose Method
S3 Method
The difference is that you do not need to set up Kinesis Data Firehose.
2-2 Lambda
We made changes to the script used in Lambda.
Until now, the script could only be used with path structures of WAF logs sent through Kinesis Data Firehose, but the new script can work with the three patterns shown below.
- Path structures of WAF logs sent through Kinesis Data Firehose
- Path structures of WAF logs sent through Kinesis Data Firehose using Hive naming convention for timestamp
- Path structures of WAF logs sent through S3
*You do not need to replace index.js if you continue to use Kinesis Data Firehose that has already been set up. If you want to, you could replace the index.js with the new one, which will work perfectly fine.
*The new index.js is compatible with Hive naming convention below:
/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/
3. Adjustments to make when switching to S3 method with existing Lambda
Below are the lists of things you don’t need and settings to adjust if you decide to keep the Lambda function you have set up in the past when switching to the S3 method.
Things you will no longer use:
- Kinesis Data Firehose
- S3 bucket used to store logs sent through Kinesis Data Firehose (if you want to keep the past logs, you do not need to delete the S3 bucket itself)
Settings you need to adjust:
-
Web ACL’s Logging and metrics configuration
-
Change the Resources of the read permission policy used by Lambda to your new WAF log destination (S3)
-
Change the S3 information set for Lambda trigger
-
Replace function code used in Lambda
*Due to browser cache, you might see the previous version of function code, so please clear the cache before obtaining the function code.
When you edit IAM permission, the change may not be applied to a running Lambda function. In that case, insert a blank line at the end of index.js and deploy again.
If you are not going to use the existing Lambda function you have been using, refer to the manual below to set up a new S3 bucket and Lambda function. Delete unnecessary Kinesis Data Firehose and the old Lambda function when the new S3 bucket and Lambda function are ready.
4. Things to keep in mind
WafCharm’s Notification feature is triggered to operate when a WAF log file is created.
With Kinesis Data Firehose, notification intervals change according to the values you set in Kinesis Data Firehose’s Buffer intervals and Buffer size setting.
However, if you publish WAF logs to S3 directly, intervals would be the default 5-minute interval defined by AWS.
Each notification email (log file) sent by WafCharm contains details of a maximum of 10 detections; this specification has not changed. By using WAF logs published to an S3 bucket, there are more chances for the number of detections to exceed the maximum number of detections the notification feature can handle compared to the Kinesis Data Firehose.
If you want to receive the notification emails for detections as much as possible, consider using Kinesis Data Firehose.
If you are switching the method in an existing environment, some WAF logs published during the change may not be transferred to WafCharm. If you’d like to lessen the number of WAF logs that are not sent to WafCharm, complete the Lambda setting first and then configure Web ACL’s Logging and metrics setting last.
5. Conclusion
If you are already using the Reporting & Notification feature, changing the setting may not be necessary, considering the effort it takes to switch methods. If you are looking to cut back on the cost of Kinesis Data Firehose usage, this could be something you’d like to try.
