This post was originally published in Japanese in the past.

Introduction

You can fully obtain logs for AWS WAF. If you are in charge of analyzing WAF logs, please take a look at this post for more details.

What is full logging?

There is a feature to obtain logs that matched a rule called GetSampledRequests. This is a useful feature, but you can only obtain part of the logs within three hours of the request. However, in the full log, you can see all of the logs.

What gets logged in full logs

In full logs, all of the access to AWS WAF will be recorded. The point here is the word "all," because even if the request is not detected, it will be logged. In other words, the request that did not match any rules and the default action was taken will be included as well. Therefore, if you only want to examine detected data, you will need to exclude the records that include such values.

Can full logs obtain POST body?

The difficult part of operating AWS WAF is false positives detected in the POST body data. In the Sampled Requests, POST Body was not included and could not investigate the false positives. Unfortunately, the POST body will not be included in full logs either. The word "full" in this case refers to the fact that all of the access that goes through WAF will be recorded.

Format

Information included in a single access

Full logs obtain logs in JSON format per access just like sampled requests logs. An example of a log of single access is as below.

{
  "timestamp"          :1534531102630,
  "formatVersion"      :1,
  "webaclId"           :"360cb717-5a9f-4f2f-ac64-09ab912af591",
  "terminatingRuleId"  :"Default_Action",
  "terminatingRuleType":"REGULAR",
  "action"             :"ALLOW",
  "httpSourceName"     :"CF",
  "httpSourceId"       :"i-123",
  "ruleGroupList"      :[
    {
      "ruleGroupId"    :"61f4eb08-4e1b-4394-92b5-e8abf834fad6",
      "terminatingRule":null,"nonTerminatingMatchingRules":[]
    }
  ],
  "rateBasedRuleList":[
    {
      "rateBasedRuleId":"7c968ef6-32ec-4fee-96cc-51198e412e7f",
      "limitKey"       :"IP",
      "maxRateAllowed" :2000
    },
    {
      "rateBasedRuleId":"4659b169-2083-4a91-bbd4-08851a9aaf74",
      "limitKey"       :"IP",
      "maxRateAllowed" :2000
    }
  ],
  "nonTerminatingMatchingRules":[
    {
      "ruleId":"7c968ef6-32ec-4fee-96cc-51198e412e7f",
      "action":"COUNT"
    }
  ],
  "httpRequest":{
    "clientIp":"192.10.23.10",
    "country" :"US",
    "headers" :[
      {"name":"Host","value":"127.0.0.1:1989"},
      {"name":"User-Agent","value":"curl/7.53.1"},
      {"name":"Accept","value":"*/*"}
    ],
    "uri"        :"REDACTED",
    "args"       :"",
    "httpVersion":"HTTP/1.1",
    "httpMethod" :"GET",
    "requestId"  :"distribution_id"
  }
}

There is information related to the request itself like source IP address, country, header, and URI under httpRequest. Other than that, there is information related to AWS WAF such as webaclId and terminatingRuleId, which tells which rule in which Web ACL detected the request.

Format of the file

As a file of these full logs, each access shown in JSON format will be concatenated and written into a file. This format follows the existing format from Kinesis Firehose.

{JSON for single access}{JSON for single access}{JSON for single access}...

However...

Here is a good news. When JSON is concatenated like above, you could not use the format directly in Athena and had to use Lambda to avoid the issue. In the full log, you can get a JSON file with line break.

The format is like below. With this update, using Athena to analyze the logs has become a lot easier.

{JSON for single access}\n{JSON for single access}\n{JSON for single access}\n...

Differences with Sampled Requests

There are no differences between the content of JSON per access. Thus, you cannot see POST body data even in full logs. Below is a list of differences between full logs and sampled requests.

How to obtain full logs

WAF Full logs can be obtained using Kinesis Firehose. You can choose the destination from the options that Firehose provides, which include S3, RedShift, and ElasticSearch. The preparation you need for each service will be done through Kinesis's features.

• Access the AWS management console
• Specify destintion path (S3, Athena, etc.)

Conclusion

We looked at the full log feature in this post. With this update, you will be able to collect and analyze the detection status even more.