This post was originally published in Japanese in the past.

Table of Contents

1. 1. Introduction
2. 2. What happens when access is blocked by AWS WAF
3. 3. Format of Application Load Balancer's access logs
4. 4. What to look for
5. 5. What you cannot see in the access logs
6. 6. Conclusion

1. Introduction

We will take a look at a way to check the blocking status of AWS WAF in Application Load Balancer’s access logs in this post.
You can see if access was blocked by AWS WAF by looking at WAF logs, but the format is a bit more complicated compared to access logs. If you are simply looking at blocked IP addresses or the number of blocked requests, access logs can provide that information.

2. What happens when access is blocked by AWS WAF

When requests are blocked by AWS WAF, HTTP status 403 (Forbidden) is returned.
Resources attached to AWS WAF will respond, so there will be no information left for the access logs in the web server.

Official information:
AWS WAF rule action

3. Format of Application Load Balancer's access logs

Important information to look for in the access logs when requests are blocked by AWS WAF is the actions_executed field.
This is a field that stores executed actions; if requests are blocked by AWS WAF, “waf” will be the last action.

Examples

Regular log

http 2020-08-12T02:14:17.638988Z app/XXXXXX/XXXXXX YYY.YYY.YYY.YYY:55048 YYY.YYY.YYY.YYY:80 0.013 0.003 0.000 200 200 475 355 "GET http://XXXXXXXXX HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" - - arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXX:targetgroup/XXXXXXX/XXXXXXXXX "Root=1-5f335079-29e10ae582d659822c5ffab0" "-" "-" 0 2020-08-12T02:14:17.623000Z "waf,forward" "-" "-" "YYY.YYY.YYY.YYY:80" "200" "-" "-"

Log of a request blocked by AWS WAF

http 2020-08-12T02:20:56.385481Z app/XXXXXX/XXXXXX YYY.YYY.YYY.YYY:55114 - -1 -1 -1 403 - 485 689 "GET http://XXXXXXXXX HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" - - arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXX:targetgroup/XXXXXXX/XXXXXXXXX "Root=1-5f335208-f62c4c12b823cb9be8b8a8d8" "-" "-" -1 2020-08-12T02:20:56.331000Z "waf" "-" "-" "-" "-" "-" "-"

Official information:
Access logs for your Application Load Balancer

4. What to look for

You can now look at access logs using key points explained in sections 2 and 3 to check which requests were blocked.

When searching, make sure to look for the requests that contain both of the information listed below.

• HTTP status code 403
• ”waf” is present in the actions_executed field

*There are cases where the last action in the actions_executed field could be “waf” even if the HTTP status code is not 403.

Below is an example of a query that checks the number of IP addresses from the target access log.
*Simply searching for 403 will match strings other than status code, so we’ve added space and hyphen.

例

$ grep " 403 - " access.log | grep '"waf"' | awk '{print $4}' | awk '{sub(":.*", "");print $0;}'  | sort | uniq  -c
   1775 XXX.XXX.XXX.XXX
      1 XXX.XXX.XXX.XXX
     48 XXX.XXX.XXX.XXX
      1 XXX.XXX.XXX.XXX
     12 XXX.XXX.XXX.XXX
     72 XXX.XXX.XXX.XXX
      1 XXX.XXX.XXX.XXX
     23 XXX.XXX.XXX.XXX
   1659 XXX.XXX.XXX.XXX

5. What you cannot see in the access logs

Access logs only document the fact that AWS WAF returned 403, so you cannot see which rule detected the attacks just by looking at the access logs.
If you need to respond to false positives, check the AWS WAF logs.

In addition, some rules are targeted to detect attacks in the body of the request, but body data are not exported in access logs.
Also, note that AWS WAF logs will not export the actual content they matched against except for certain rules provided by AWS. In such a case, you might need to ask what the users did and test the steps to replicate the request or change the actions of the matched rule to COUNT to export POST content on web servers.

6. Conclusion

You can probably see that Application Load Balancer’s access logs can provide basic information regarding requests blocked by AWS WAF.
However, if you are going to respond to false positives, you will need to understand the structure of WAF logs as well.

In cases where you need to check the status regularly or in real-time, we recommend using Amazon Elasticsearch Service.
Analyze AWS WAF log using Amazon’s Elasticsearch Service
(This post shows an example of using Amazon Elasticsearch Service with AWS WAF Classic.)

*Access logs can be exported from CloudFront and API Gateway as well. To narrow down, look at a combination of HTTP status code 403, x-edge-result-type, and x-edge-detailed-result-type fields for CloudFront and a combination of HTTP status code 403 and context.wafResponseCode field for API Gateway.