This post is always in progress This blog post was initially written on 2021 and AWS continues to add features continuously. If you notice something wrong or obsolete, share it to me!
Why this repository
AWS lets customers to generate and integrate some services security logs. However, each log source has its own configuration making understanding quite difficult for new customers.
This post references main AWS services logging sources with its properties, format(s), some examples and resource to go further.
Services grouped by log types
- API Calls
- Network
- Web access
- Security & compliance
- Other applications
Help - Repository guide Property definitions are available in the last section.
CloudTrail logs
CloudTrail logs most of API call events made on AWS Control plane and few ones on Data plane.
There are 3 methods to record CloudTrail events described below:
| Log | CloudTrail Event History |
|---|---|
| Content | Last 90 days of AWS API call details for Management events. It’s enabled by default and cannot be disabled.Included events types: Read and Write Management event (control plane). Formerly known as API Activity History. |
| Format | JSON |
| Delivery | Between 1 and 15 min (smaller delivery time since September 2021) |
| Output | Only available trough the Console or API (must be queried) |
| Custom/Filter | Console: apply only one filter at a time |
| Scope | Account |
| ID | None |
| Sharing | No |
| Regional | Yes. IAM Service events in us-east-1 only |
| Cost | Free |
| Availability | API Activity History: Mars 2015 (last 7 days on limited services and regions).Event History: August 2017 (GA, all regions), June 2018 (All Management events and last 90 days) |
| Log | CloudTrail trail |
|---|---|
| Content | A trail records most of AWS API call details.Events types can be selected: Read and Write Management events (Control plane), Data events (Data plane) or Insights (unusual volume of management events) |
| Format | JSON |
| Delivery | 15 min |
| Output | S3 bucket (objects name format AccountID_CloudTrail_RegionName_YYYYMMDDTHHmmZ_UniqueString.json.gz) and/or a CloudWatch Logs log group |
| Custom/Filter | Event type (Management, Data, Insight), Action type (Read, Write) and region (Current, All). KMS events can be excluded. A Organization trail can be created on the Management Account applied on all Accounts and regions. |
| Scope | Trail per Account or per Organization (current and future Accounts) |
| Regional | Yes. IAM Service events in us-east-1 only |
| ID | ARN: arn:${Partition}:cloudtrail:${Region}:${Account}:trail/${TrailName} |
| Sharing | No |
| Cost | The first trail of Management events is free, the following are paying as well as other events types: Data and Insights. Indirect charges: CloudWatch/S3 charges |
| Availability | November 2013 (GA), November 2016 (S3 Data Events), November 2019 (Insights) |
| Log | EventBridge default bus |
|---|---|
| Content | AWS API call details (recorded by CloudTrail) can be caught in the EventBridge default event bus of the Account with a rule. Only Write Management event (control plane) are available. EventBridge is formerly known as CloudWatch Events (same API with improvements). |
| Format | JSON |
| Delivery | Near-real time (1 sec) |
| Output | EventBridge Rule target |
| Custom/Filter | Highly granular, any JSON event pattern |
| Scope | Account |
| ID | ARN: arn:${Partition}:events:${Region}:${Account}:rule/[${EventBusName}/]${RuleName} |
| Sharing | Yes by using another Account bus as Rule target |
| Regional | Yes. IAM Service events in us-east-1 only |
| Cost | Free. Indirect: EventBridge rule target charges |
| Availability | January 2016 |
Format
All CloudTrail logs are generated in JSON. Content could be complex to parse because JSON structure depends of each service and action details (What). However details about the Principal (Who) and Request Context (When and How) have the same structure.
AWS documents the record contents.
Samples
- Sign-in events on Console
All these events are recorded with "eventType": "AwsServiceEvent".
Existing IAM User - Successful MFA Console Login
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAXY6RSLOFBLXXXXXXX",
"arn": "arn:aws:iam::111111111111:user/user-name-1",
"accountId": "111111111111",
"userName": "user-name-1"
},
"eventTime": "2020-03-29T13:49:13Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.2.3.4",
"userAgent": "Mozilla/5.0 ... Firefox/73.0",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Success"
},
"additionalEventData": {
"LoginTo": "https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true",
"MobileVersion": "No",
"MFAUsed": "Yes"
},
"eventID": "45fadea2-caac-4bd4-859a-48a9762c66a5",
"eventType": "AwsConsoleSignIn",
"recipientAccountId": "111111111111"
}
Existing IAM User - Failed Console Login
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAXY6RSLOFBLXXXXXXX",
"accountId": "111111111111",
"accessKeyId": "",
"userName": "user-name-1"
},
"eventTime": "2020-03-29T15:53:13Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.2.3.4",
"userAgent": "Mozilla/5.0 ... Firefox/73.0",
"errorMessage": "Failed authentication",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Failure"
},
"additionalEventData": {
"LoginTo": "https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true",
"MobileVersion": "No",
"MFAUsed": "Yes"
},
"eventID": "ee74b50a-1272-4e87-907f-e6094bd00e84",
"eventType": "AwsConsoleSignIn",
"recipientAccountId": "111111111111"
}
No-existing IAM User - Failed Console Login
Therefore AWS replace the username with HIDDEN_DUE_TO_SECURITY_REASONS.
{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"accountId": "111111111111",
"accessKeyId": "",
"userName": "HIDDEN_DUE_TO_SECURITY_REASONS"
},
"eventTime": "2020-03-29T15:37:18Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.2.3.4",
"userAgent": "Mozilla/5.0 ... Firefox/73.0",
"errorMessage": "No username found in supplied account",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Failure"
},
"additionalEventData": {
"LoginTo": "https://console.aws.amazon.com/console/home?state=hashArgs%23&isauthcode=true",
"MobileVersion": "No",
"MFAUsed": "No"
},
"eventID": "c053eda0-fda0-4bd2-b9e9-cc9517556f8a",
"eventType": "AwsConsoleSignIn",
"recipientAccountId": "111111111111"
}
Successful Console Login with STS tokens
Federation Console link is generated with STS temporary tokens.
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AIDAXY6RSLOFBLXXXXXXX:Session_AWS-CLI_1234",
"arn": "arn:aws:sts::111111111111:assumed-role/user-name-2/Session_AWS-CLI_1234",
"accountId": "111111111111",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2020-03-26T21:58:31Z"
},
"sessionIssuer": {
"type": "Role",
"principalId": "AIDAXY6RSLOFBLXXXXXXX",
"arn": "arn:aws:iam::111111111111:role/user-name-2",
"accountId": "111111111111",
"userName": "user-name-2"
}
}
},
"eventTime": "2020-03-26T22:02:47Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.2.3.4",
"userAgent": "Mozilla/5.0 ... Firefox/73.0",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Success"
},
"additionalEventData": {
"MobileVersion": "No",
"MFAUsed": "No"
},
"eventID": "cb31c487-7527-4540-8b62-d2e294c5db83",
"eventType": "AwsConsoleSignIn",
"recipientAccountId": "111111111111"
}
AWS SSO - Successful Console Login
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAZZTXDSP5VXXXXXXXX:sso-username-1",
"arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_sso-role_720f8b25b6806XXX/sso-username-1",
"accountId": "111111111111",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAZZTXDSP5VXXXXXXXX",
"arn": "arn:aws:iam::111111111111:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_sso-role_720f8b25b6806XXX",
"accountId": "111111111111",
"userName": "AWSReservedSSO_sso-role_720f8b25b6806XXX"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2021-10-28T20:13:23Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2021-10-28T20:13:23Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "1.2.3.4",
"userAgent": "Mozilla/5.0 ... Chrome/95.0.4638.54 Safari/537.36",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Success"
},
"additionalEventData": {
"MobileVersion": "No",
"MFAUsed": "No"
},
"eventID": "d09430a8-d1be-4d72-9a22-c833817e5cd0",
"readOnly": false,
"eventType": "AwsConsoleSignIn",
"managementEvent": true,
"recipientAccountId": "111111111111",
"eventCategory": "Management"
}
- Others
Cloudwatch Log stream created by the CloudWatch Agent on EC2 instance
The EC2 instance use his instance role to push logs to CloudWatch Logs ; the first time a Log Stream is created.
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROA4HGNPFE42IQKZGFHF:i-0bb6317a82af461f3",
"arn": "arn:aws:sts::111111111111:assumed-role/ec2_CloudWatchAgent/i-0bb6317a82af461f3",
"accountId": "111111111111",
"accessKeyId": "ASIA4HGNPFE4UFRDB77L",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROA4HGNPFE42IQKZGFHF",
"arn": "arn:aws:iam::111111111111:role/ec2_CloudWatchAgent",
"accountId": "111111111111",
"userName": "ec2_CloudWatchAgent"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2021-10-31T12:54:25Z",
"mfaAuthenticated": "false"
},
"ec2RoleDelivery": "2.0"
}
},
"eventTime": "2021-10-31T13:19:39Z",
"eventSource": "logs.amazonaws.com",
"eventName": "CreateLogStream",
"awsRegion": "us-east-1",
"sourceIPAddress": "54.91.187.20",
"userAgent": "CWAgent/1.247347.4 (go1.15.8; linux; arm64) 2021-02-24T00:14:10+0000 inputs:(mem disk logfile) outputs:(cloudwatch cloudwatchlogs)",
"requestParameters": {
"logGroupName": "instance-log",
"logStreamName": "i-0bb6317a82af461f3"
},
"responseElements": null,
"requestID": "44b30da7-ec0c-4c98-b4d0-f77606665c41",
"eventID": "403e18e9-555f-4bdb-9f2e-e5af82e47a4d",
"readOnly": false,
"eventType": "AwsApiCall",
"apiVersion": "20140328",
"managementEvent": true,
"recipientAccountId": "111111111111",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "logs.us-east-1.amazonaws.com"
}
}
Resources
API Gateway - REST API
https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-logging.html
API Gateway - HTTP API
https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-logging.html
Route 53 Resolver DNS queries
| Log | Route 53 Resolver Query Logging |
|---|---|
| Content | All DNS queries received by a VPC DNS resolver |
| Format | JSON |
| Delivery | 10-20 sec |
| Output | CloudWatch Logs group, S3 bucket or Kinesis Data Firehose delivery stream |
| Custom/Filter | No |
| Scope | VPC |
| Regional | Yes |
| ID | ARN: arn:aws:route53resolver:<region>:<acountID>:resolver-query-log-config/rqlc-<16digit> |
| Sharing | Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager |
| Cost | Free. Indirect: CloudWatch/S3/Kinesis charges |
| Availability | August 2020 |
The Resolver Query Logging Configuration can only have one output and cannot be edited after creation.
Format
See below samples, JSON structure details format.
Sample
Real samples with a hosted private zone mhg.vpc and a public zone.
Private zone - Existing record
``` json
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T09:25:02Z",
"query_name": "asset1.mhg.vpc.",
"query_type": "A",
"query_class": "IN",
"rcode": "NOERROR",
"answers": [
{
"Rdata": "10.5.0.34",
"Type": "A",
"Class": "IN"
}
],
"srcaddr": "172.31.24.160",
"srcport": "41366",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
```
Private zone - No-existing record
``` json
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T09:32:05Z",
"query_name": "test2.mhg.vpc.",
"query_type": "A",
"query_class": "IN",
"rcode": "NXDOMAIN",
"answers": [],
"srcaddr": "172.31.24.160",
"srcport": "37654",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
```
Private zone - Existing public record
``` json
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T10:06:16Z",
"query_name": "mhg.ovh.",
"query_type": "A",
"query_class": "IN",
"rcode": "NOERROR",
"answers": [
{
"Rdata": "13.32.145.91",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.81",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.126",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.44",
"Type": "A",
"Class": "IN"
}
],
"srcaddr": "172.31.24.160",
"srcport": "58050",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
```
Route 53 Public DNS queries
| Log | Public DNS queries (public zone) |
|---|---|
| Content | DNS queries of the NS servers (resolvers, not final clients) |
| Format | Line with values separated by space |
| Delivery | 2-3 min |
| Output | CloudWatch Logs group |
| Custom/Filter | No |
| Scope | Route 53 public zone |
| Regional | No. Stored in us-east-1 only |
| ID | UUIDv4 (no ARN) |
| Sharing | Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager |
| Cost | Free. Indirect: CloudWatch Logs charges |
| Availability | September 2017 |
CloudWatch Logs allows you to export streams to a S3 bucket, however you have to call yourself the CloudWatch API to initiate this asynchronous task. a scheduled Lambda function can be used to do the job.
Following samples come from S3 and includes the CloudWatch Logs timestamp because the Export Task API call task adds it in the S3 object.
Format
CloudWatch-timestamp Log-format-version Query-timestamp Hosted-zone-ID Query-name Query-type Response-code Layer-4-protocol Route-53-edge-location Resolver-IP-address EDNS-client-subnet
Sample
AAAA DNS request on mhg.ovh
2020-03-06T00:31:09.000Z 1.0 2020-03-06T00:31:09Z Z3M111CF8036L2 mhg.ovh AAAA NOERROR UDP VIE50 37.187.141.25 -
Flow Logs
| Log | Flow Logs |
|---|---|
| Content | IP Flow metadatas (ISO Layer 4) similar to NetFlow or IPFIX. |
| Format | Line with values separated by space |
| Delivery | 1 or 10 minutes (aggregation interval) |
| Output | CloudWatch Logs, S3 Bucket |
| Custom/Filter | Custom format available (fields and order) |
| Scope | VPC, Subnet or network interface (ENI). |
| Regional | Yes |
| ID | fl-<17charcaters> (No ARN) |
| Sharing | No |
| Cost | Free. Indirect: CloudWatch Logs/S3 charges |
| Availability | June 2015 (GA), August 2018 (S3 target) |
Format
Custom format with all fields
${version} ${account-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${type} ${srcaddr} ${dstaddr} ${pkt-srcaddr} ${pkt-dstaddr} ${srcport} ${dstport} ${protocol} ${tcp-flags} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
Samples
Custom format - Accept and Reject streams
Version number is 3 because of custom format.
Accepted:
3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 52.95.155.48 172.31.40.102 52.95.155.48 172.31.40.102 443 37318 6 19 15577 21652528 1577191414 1577191424 ACCEPT OK
Rejected:
3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 216.218.206.85 172.31.40.102 216.218.206.85 172.31.40.102 52862 5900 6 2 1 40 1577191398 1577191404 REJECT OK
CloudFront Access Logs
| Log | Cloudfront standard logs |
|---|---|
| Content | Web/RTMP access log of the CDNs similar to reverse proxy logs. |
| Format | Line with values separated by space |
| Delivery | Up to several times an hour. Can sometimes be delayed by up to 24 hours |
| Output | S3 Bucket |
| Custom/Filter | No |
| Scope | Distribution |
| Regional | No, stored in us-east-1 only |
| ID | None |
| Sharing | ? |
| Cost | Free. Indirect: S3 charges |
| Availability | May 2009 (GA), December 2019 (new fields) |
| Log | Real-time log configuration |
|---|---|
| Content | Web/RTMP access log of the CDN, similar to reverse proxy logs. |
| Format | Line |
| Delivery | Near real-time (1 sec) |
| Output | Kinesis data stream |
| Custom/Filter | Sampling rate (1-100%), fields (among 40) and Cache behavior |
| Scope | Attach configuration to Distribution(s) |
| Regional | No, stored in us-east-1 only |
| ID | ARN: arn:aws:cloudfront::<accountID>:realtime-log-config/<configID> |
| Sharing | To an external Kinesis data stream |
| Cost | Volume of log. Indirect: Kinesis |
| Availability | August 2020 |
Format
Standard log
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
Sample
Standard log
2019-12-26 09:12:26 FRA50-C1 3198 1.2.3.4 GET djm1t0idtm8px.cloudfront.net /about.html 200 https://mhg.ovh/ Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010.15;%2520rv:71.0)%2520Gecko/20100101%2520Firefox/71.0 - - Miss BCFMyYSZyL1u6ZixT2CLGNnFBcVg7P2Y-Rr93r68KwxN2d6OoFyVfw== mhg.ovh https 253 0.078 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0 - - 50024 0.078 Miss text/html - - -
GuardDuty findings
| Log | GuardDuty finding S3 Export |
|---|---|
| Content | Security alerts |
| Format | JSON (details) |
| Delivery | 5 min for new finding and 15min/1h/6h for next occurrences of the same issue (finding aggregation) |
| Output | S3 bucket with a mandatory KMS CMK key. |
| Custom/Filter | No |
| Scope | GuardDuty Detector |
| Regional | Yes |
| ID | 32 characters (no ARN) |
| Sharing | Possible to a external and unique S3 bucket for multiple Export configuration |
| Cost | Free. Indirect: S3 and KMS CMK charges |
| Availability | November 2019 |
| Log | GuardDuty finding through EventBridge |
|---|---|
| Content | GuardDuty findings (Security alerts) can be caught in the EventBridge default event bus of the Account with a rule. |
| Format | JSON (details) |
| Delivery | 5 min for new finding and 15min/1h/6h for next occurrences of the same issue (finding aggregation) |
| Output | EventBridge event bus (Account default or custom) |
| Custom/Filter | Rule pattern |
| Scope | Account (GuardDuty Detector) |
| Regional | Yes |
| ID | ARN: arn:${Partition}:events:${Region}:${Account}:rule/[${EventBusName}/]${RuleName} |
| Sharing | Possible with Rule target |
| Cost | Free. Indirect: EventBridge |
| Availability | November 2017 |
Remarks
- If you use the Multi-Account/Organization feature in a GuardDuty administrator account, all findings related to member Accounts will be exported to the S3 bucket configured in the Administrator account.
Format
Please refer to the AWS documentation.
Samples
IAMUser/RootCredentialUsage finding
Policy:IAMUser/RootCredentialUsage finding: An API was invoked using root credentials.
{
"schemaVersion": "2.0",
"accountId": "111111111111",
"region": "us-east-1",
"partition": "aws",
"id": "aab8f74669d66b7115893a8a87779XXX",
"arn": "arn:aws:guardduty:eu-west-3:111111111111:detector/caFC17dEAbDBCe5CA3EeEeEBaEXXXXXXX/finding/aab8f74669d66b7115893a8a8XXXXXXX",
"type": "Policy:IAMUser/RootCredentialUsage",
"resource":
{
"resourceType": "AccessKey",
"accessKeyDetails":
{
"accessKeyId": "ASIA2MSOSYTBPZMUXXXX",
"principalId": "111111111111",
"userType": "Root",
"userName": "Root"
}
},
"service":
{
"serviceName": "guardduty",
"detectorId": "caFC17dEAbDBCe5CA3EeEeEBaEEcEXXX",
"action":
{
"actionType": "AWS_API_CALL",
"awsApiCallAction":
{
"api": "CreateServiceLinkedRole",
"serviceName": "iam.amazonaws.com",
"callerType": "Remote IP",
"remoteIpDetails":
{
"ipAddressV4": "72.21.217.15",
"organization":
{
"asn": "16509",
"asnOrg": "AMAZON-02",
"isp": "Amazon.com",
"org": "Amazon.com"
},
"country":
{
"countryName": "United States"
},
"city":
{
"cityName": "Ashburn"
},
"geoLocation":
{
"lat": 39.0481,
"lon": -77.4728
}
},
"affectedResources":
{}
}
},
"resourceRole": "TARGET",
"additionalInfo":
{},
"evidence": null,
"eventFirstSeen": "2020-05-07T09:34:32Z",
"eventLastSeen": "2020-05-07T09:35:35Z",
"archived": false,
"count": 2
},
"severity": 2,
"createdAt": "2020-05-07T09:52:18.860Z",
"updatedAt": "2020-05-07T09:52:18.860Z",
"title": "API CreateServiceLinkedRole was invoked using root credentials.",
"description": "API CreateServiceLinkedRole was invoked using root credentials from IP address 72.21.217.15."
}
Security Hub
| Log | Security Hub finding trough EventBridge |
|---|---|
| Content | Alerts received from other security tools |
| Format | JSON, AWS Security Finding Format (ASFF) |
| Delivery | Near real-time (1sec) ? |
| Output | EventBridge Account default bus |
| Custom/Filter | Rule pattern and 3 event types: All findings, Findings for custom actions, Insight results for custom actions |
| Scope | Security Hub |
| Regional | Yes |
| ID | EventBridge |
| Sharing | Another Account with EventBridge target |
| Cost | Free. Indirect: EventBridge |
| Availability | November 2018 |
Format
AWS published a standardized format, AWS Security Finding Format (ASFF), detailed on the documentation.
Samples
GuardDuty findings - S3.BucketAnonymousAccessGranted
{
"SchemaVersion": "2018-10-08",
"Id": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
"ProductArn": "arn:aws:securityhub:eu-west-3::product/aws/guardduty",
"GeneratorId": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX",
"AwsAccountId": "111122223333",
"Types": [
"Software and Configuration Checks/Policy:S3.BucketAnonymousAccessGranted"
],
"FirstObservedAt": "2020-08-11T17:09:59Z",
"LastObservedAt": "2020-08-11T17:09:59Z",
"CreatedAt": "2020-08-11T17:22:57.121Z",
"UpdatedAt": "2020-08-11T17:22:57.121Z",
"Severity": {
"Product": 8,
"Label": "HIGH",
"Normalized": 80
},
"Title": "Amazon S3 Public Anonymous Access was granted for S3 bucket bucket1.",
"Description": "The Amazon S3 bucket bucket1 was granted public anonymous access by Role1 calling PutBucketPolicy. If this behavior is not expected, it may indicate a configuration mistake or that your credentials are compromised.",
"SourceUrl": "https://eu-west-3.console.aws.amazon.com/guardduty/home?region=eu-west-3#/findings?macros=current&fId=94b9ef45f2f0f4b2dc00006d77b8XXXX",
"ProductFields": {
"aws/guardduty/service/archived": "false",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asnOrg": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/org": "XXXX",
"aws/guardduty/service/additionalInfo": "",
"aws/guardduty/service/resourceRole": "TARGET",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/isp": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lat": "X.X",
"aws/guardduty/service/count": "1",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/ipAddressV4": "1.1.1.1",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/country/countryName": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/callerType": "Remote IP",
"aws/guardduty/service/action/awsApiCallAction/serviceName": "s3.amazonaws.com",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/city/cityName": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/api": "PutBucketPolicy",
"aws/guardduty/service/serviceName": "guardduty",
"aws/guardduty/service/evidence": "",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lon": "X.X",
"aws/guardduty/service/detectorId": "fcb76996ec7afe8c7fbef8e5c08eXXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asn": "XXXX",
"aws/guardduty/service/eventFirstSeen": "2020-08-11T17:09:59Z",
"aws/guardduty/service/action/awsApiCallAction/affectedResources/AWS::S3::Bucket": "bucket1",
"aws/guardduty/service/eventLastSeen": "2020-08-11T17:09:59Z",
"aws/guardduty/service/action/actionType": "AWS_API_CALL",
"aws/securityhub/FindingId": "arn:aws:securityhub:eu-west-3::product/aws/guardduty/arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
"aws/securityhub/ProductName": "GuardDuty",
"aws/securityhub/CompanyName": "Amazon"
},
"Resources": [
{
"Type": "AwsIamAccessKey",
"Id": "AWS::IAM::AccessKey:ASIA4HGNPFE47Q4MXXXX",
"Partition": "aws",
"Region": "eu-west-3",
"Details": {
"AwsIamAccessKey": {
"PrincipalId": "AROA4HGNPFE4XI5FLXXXX:user1",
"PrincipalType": "AssumedRole",
"PrincipalName": "Role1"
}
}
}
],
"WorkflowState": "NEW",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE"
}
Resources
S3 logs
Multiple methods exist to record events on Simple Storage Service (S3):
| Log | Server access logging |
|---|---|
| Content | S3 Access logs records every call made on a S3 Bucket, including unauthenticated/anounimous access. It’s similar to Web server logs. |
| Format | Line with values separated by space (details) |
| Delivery | Few hours or less |
| Output | S3 bucket: the bucket source itself (not recommended) or any one the same Account and Region. File name frmat: TargetPrefixYYYY-mm-DD-HH-MM-SS-UniqueString/ |
| Custom/Filter | No |
| Scope | S3 bucket |
| Regional | Yes |
| ID | None |
| Sharing | No |
| Cost | Free. Indirect: S3 |
| Availability | Before 2009 (GA), March 2019 (6 new fields) |
| Log | S3 CloudTrail Data events |
|---|---|
| Content | CloudTrail can API call details made on bucket-level (Management event) and/or object-level (Data event).Please refer to the CloudTrail chapter for all details. |
| Format | - |
| Delivery | - |
| Output | - |
| Custom/Filter | - |
| Scope | - |
| Regional | - |
| ID | - |
| Sharing | - |
| Cost | - |
| Availability | - |
AWS recommends that you use AWS CloudTrail data events instead of Amazon S3 access logs. CloudTrail data events are easier to set up and contain more information. Source
Format
Please refer to the AWS documentation
Samples
CloudTrail Data event - Athena reads an object
Object event trough Cloudtrail: Athena Service reads an object on behalf of an IAM User that assumed a Role in the Account’s bucket. The example show a bucket with Scott Piper’s flaws.cloud CloudTrail logs.
{
"eventVersion": "1.07",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROA4HGNPFE4XI5FLXXXX:user1",
"arn": "arn:aws:sts::111122223333:assumed-role/OrganizationAccountAccessRole/user1",
"accountId": "111122223333",
"accessKeyId": "ASIAIAH3T5UEST3NXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROA4HGNPFE4XI5FLXXXX",
"arn": "arn:aws:iam::111122223333:role/Role1",
"accountId": "111122223333",
"userName": "Role1"
},
"attributes": {
"creationDate": "2020-10-31T16:41:18Z",
"mfaAuthenticated": "true"
}
},
"invokedBy": "athena.amazonaws.com"
},
"eventTime": "2020-10-31T17:05:36Z",
"eventSource": "s3.amazonaws.com",
"eventName": "GetObject",
"awsRegion": "us-east-1",
"sourceIPAddress": "athena.amazonaws.com",
"userAgent": "athena.amazonaws.com",
"requestParameters": {
"bucketName": "xxxx-flaws-cloud-cloudtrail",
"Host": "xxxx-flaws-cloud-cloudtrail.s3.amazonaws.com",
"key": "flaws_cloudtrail09.json.gz"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"bytesTransferredIn": 0,
"AuthenticationMethod": "AuthHeader",
"x-amz-id-2": "YVT7gnHZDu7agzbOV+t8GECNPiJ31IErBUGuTEFkJyxvQ4HYylq5w0SBQOogF9u9Q+iee54wXXXX",
"bytesTransferredOut": 10977945f
},
"requestID": "DE670E80ED3590E0",
"eventID": "b02731a7-41b4-4cd5-aabf-858d447bXXXX",
"readOnly": true,
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail/flaws_cloudtrail09.json.gz"
},
{
"accountId": "111122223333",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail"
}
],
"eventType": "AwsApiCall",
"managementEvent": false,
"recipientAccountId": "111122223333",
"eventCategory": "Data"
}
WAF logs
TODO
Application Load Balancer
TODO
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html
Network Load Balancer
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html
RDS
TODO
VPC Lattice
| Log | VPC Lattice service network / service access log |
|---|---|
| Content | HTTP(S) access logs of VPC Lattice service network or service. |
| Format | JSON |
| Delivery | Between 2 and 6 min (smaller after first requests in CloudWatch) |
| Output | CloudWatch Logs group, S3 bucket or Kinesis Data Firehose |
| Custom/Filter | No |
| Scope | Service network (all linked services) or individual service |
| ID | None |
| Sharing | No |
| Regional | Yes |
| Cost | Free. Indirect: CloudWatch/S3/Kinesis charges |
| Availability | March 2023 (service GA) |
Format
AWS documentation details JSON content.
Samples
VPC Lattice - Anonymous access allowed
The log below was sent to CloudWatch Logs and is for an allowed HTTPS request from an EC2 instance to a Lambda function in the same region and the same AWS account.
The auth policy (resource-based policy) was set as None on the service network.
The log recorded at the service network or service level is identical (tested).
{
"startTime": "2023-04-01T11:16:05Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 200,
"bytesReceived": 283,
"bytesSent": 2281,
"duration": 290,
"userAgent": "-",
"hostHeader": "lambda",
"targetIpPort": "44.192.251.238:443",
"targetGroupArn": "arn:aws:vpc-lattice:us-east-1:840097016121:targetgroup/tg-040ede59f30d6a02a",
"sourceIpPort": "172.31.30.232:60894",
"serverNameIndication": "s-test-1-01ede641cc872f98c.7d67968.vpc-lattice-svcs.us-east-1.on.aws",
"sourceVpcId": "vpc-e532159f",
"destinationVpcId": "-",
"serviceArn": "arn:aws:vpc-lattice:us-east-1:840097016121:service/svc-01ede641cc872f98c",
"serviceNetworkArn": "arn:aws:vpc-lattice:us-east-1:840097016121:servicenetwork/sn-0bd03117fade6adb5",
"requestToTargetDuration": 9,
"responseFromTargetDuration": 0,
"sslCipher": "ECDHE-RSA-AES128-GCM-SHA256",
"tlsVersion": "TLSv1.2",
"resolvedUser": "-",
"authDeniedReason": "-"
}
VPC Lattice - Anonymous access denied
The log below was sent to CloudWatch Logs and is for an denied HTTPS request from an EC2 instance to a Lambda function in the same region and the same AWS account.
The auth policy (resource-based policy) was set as IAM on the service network and allows only IAM authenticated access.
{
"startTime": "2023-04-01T15:48:06Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/2",
"responseCode": 502,
"bytesReceived": 81,
"bytesSent": 219,
"duration": 114,
"userAgent": "curl/7.88.1",
"hostHeader": "s-test-2-03ba044479164d14a.7d67968.vpc-lattice-svcs.us-east-1.on.aws",
"targetIpPort": "-",
"targetGroupArn": "arn:aws:vpc-lattice:us-east-1:111122223333:targetgroup/tg-040ede59f30d6a02a",
"sourceIpPort": "172.31.30.232:58938",
"serverNameIndication": "s-test-2-03ba044479164d14a.7d67968.vpc-lattice-svcs.us-east-1.on.aws",
"sourceVpcId": "vpc-e532159f",
"destinationVpcId": "-",
"serviceArn": "arn:aws:vpc-lattice:us-east-1:111122223333:service/svc-03ba044479164d14a",
"serviceNetworkArn": "arn:aws:vpc-lattice:us-east-1:111122223333:servicenetwork/sn-0bd03117fade6adb5",
"requestToTargetDuration": 0,
"responseFromTargetDuration": 0,
"sslCipher": "ECDHE-RSA-AES128-GCM-SHA256",
"tlsVersion": "TLSv1.2",
"resolvedUser": "Anonymous",
"authDeniedReason": "Network"
}
VPC Lattice - Authenticated access allowed
The log below was sent to CloudWatch Logs and is for an authenticated (signed) HTTPS request from an EC2 instance to a Lambda function in the same region and the same AWS account.
The auth policy (resource-based policy) was set as IAM on the service network and allows only IAM authenticated access.
{
"startTime": "2023-04-01T16:51:29Z",
"requestMethod": "GET",
"requestPath": "/",
"protocol": "HTTP/1.1",
"responseCode": 502,
"bytesReceived": 81,
"bytesSent": 1968,
"duration": 105,
"userAgent": "python-requests/2.25.1",
"hostHeader": "s-test-2-03ba044479164d14a.7d67968.vpc-lattice-svcs.us-east-1.on.aws",
"targetIpPort": "-",
"targetGroupArn": "arn:aws:vpc-lattice:us-east-1:111122223333:targetgroup/tg-040ede59f30d6a02a",
"sourceIpPort": "172.31.30.232:56822",
"serverNameIndication": "s-test-2-03ba044479164d14a.7d67968.vpc-lattice-svcs.us-east-1.on.aws",
"sourceVpcId": "vpc-e532159f",
"destinationVpcId": "-",
"serviceArn": "arn:aws:vpc-lattice:us-east-1:111122223333:service/svc-03ba044479164d14a",
"serviceNetworkArn": "arn:aws:vpc-lattice:us-east-1:111122223333:servicenetwork/sn-0bd03117fade6adb5",
"requestToTargetDuration": 0,
"responseFromTargetDuration": 0,
"sslCipher": "ECDHE-RSA-AES128-GCM-SHA256",
"tlsVersion": "TLSv1.2",
"resolvedUser": "arn:aws:sts::111122223333:assumed-role/ec2_SSM_SessionManager/i-021dd3f494cca8b54",
"authDeniedReason": "-"
}
IAM Access Analyzer
TO DO
Other types of logs (out of scope)
AWS provides tools to integrate with your own workloads. These capabilities are categorized as out of scope because it’s not generated by an AWS service and may not be security event logs.
EC2 instances logs with SSM agent / CloudWatch Log Agent
SSM agent or CloudWatch Agent can collect specific log file(s) from EC2 instances and on-premises servers, running either Linux or Windows Server ; the agent software is available on GitHub.
The installed agent natively streams log(s) content to CloudWatch Logs in near-real time (2-5 sec). Agent configuration file allows to select specific files.
Help - Repository guide
Summary table description (template)
For each Service, the following summary will give you a clear overview of the log, its capabilities and limits.
| Log | Log source name (with link to the doc) |
|---|---|
| Content | Nature of event |
| Format | File format, value separator, possible custom format |
| Delivery | Frequency of sending logs |
| Output | Where we can stream log to another service/target |
| Filter | Possibility to filter log content |
| Scope | AWS context of regarding logs |
| Regional | AWS Region linking of the log source |
| ID | The Amazon Resource Name (ARN) or ID format of the log resource (if exists) |
| Sharing | Native sharing capabilities with other AWS Accounts |
| Cost | Direct and indirect cost (without price) |
| Availability | Launch date |