AWS Systems Manager (SSM) in combination with AWS Key Management Services (KMS), Amazon CloudWatch, and Amazon OpenSearch allow administrators to encrypt and securely store user session logs, as well as search the log data for information. These tools are easy to integrate and provide powerful analytical capabilities without the undifferentiated heavy lifting.

In this series, we share the procedure for configuring AWS SSM Session Manager to securely store the session logs in Amazon CloudWatch, generate relevant metrics from the logs, and search through the logs for specific events. In Part 1, we will show you how to create a Customer Managed Key (CMK) in Amazon KMS, configure necessary permissions for usage of the keys, create a secure log group in Amazon CloudWatch, and securely send session logs to Amazon CloudWatch. In Part 2, we will query the logs in Amazon CloudWatch, create custom metrics and metric namespaces, and finally show you how to configure the delivery of the logs to Amazon OpenSearch for advanced analytics.

Background

Session Manager is an AWS SSM capability that lets you manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, on-premises instances, and virtual machines (VMs) through an interactive one-click browser-based shell, or through the AWS Command Line Interface (CLI). AWS SSM provides secure instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Traditionally system administrators use underlying operating system capabilities to log SSH session activities locally to a log file, and then have another mechanism to export this file if needed for investigation. AWS SSM Session Manager can automate the process of logging sessions, as well as transporting and storing the session logs in an encrypted central log store in Amazon CloudWatch. Amazon CloudWatch provides powerful capabilities to extract metrics from logs collected, along with sharing the information only to personnel with the right permissions levels.

Solution overview

The solution uses AWS Identity and Access Manager (IAM) roles to provide the right permissions levels to access the console logs. AWS KMS is used to create CMKs that encrypt the console logs. The solution also configures a secure log group in Amazon CloudWatch to store the Session Manager console logs.

Some important notes about this solution:

  • This solution uses AWS KMS single-Region key, which is stored and used only in the Region in which it was created. The Multi-Region key option can be selected if your implementation requires the key to be accessible in multiple-regions. However, this may entail additional configuration and steps that are not discussed in this post.

Administrator accessing EC2 instance using AWS SSM Session Manager by assuming an IAM role. Logs sent to Amazon CloudWatch and encrypted using AWS KMS.

Prerequisites

Before starting, you will need the following prerequisites:

  1. You must already be using AWS SSM. If not, then you can use our Quick Setup process to rapidly create the required IAM roles and configure the service.
  2. Each EC2 instance or on-premises server that you wish to access through Session Manager must have the SSM Agent installed and properly configured. Many of the base Amazon Machine Images (AMIs) come with this agent pre-installed. See this link for details about this configuration.
  3. The IAM role used by your instance profile must include either the AmazonSSMManagedInstanceCore managed policy, or equivalent permissions to what this would provide.
  4. Select a name for your new CloudWatch Logs log group in advance, ensuring that your log group name must be unique.

With these prerequisites met, we can proceed.

Step one: Create Amazon KMS key and policies

First, we must create a new CMK within Amazon KMS. Using a CMK to protect your session logs is required to enable certain highly sensitive data obfuscation tasks, such as masking keyboard strokes during password reset operations. Note that the creation of a KMS key will incur a small charge on your account. You can find more about pricing for Amazon KMS here

The use of KMS encryption for session log data is a best practice.

From the KMS console, first select CMKs, and then Create key.

Options available on CMKs section of Amazon KMS with Create key button highlighted

Next, specify that this is a symmetric key, and use KMS for the key material origin. Note that some customers may use Cloud Hardware Security Module (HSM) or import their own key material. In that case, they can proceed accordingly. The regionality is selected to be Single-Region key.

Configure options of CMK creation showing symmetric key selected, KMS as key material origin, and Single region key selected as regionality

Enter a name for your new CMK. We will use “session-manager-logs” for this solution. Alternatively, you can use a name that is meaningful to you and your organization. Click Next.

Specifying key alias and description as part of Step 2 of CMK creation

Now, the CMK creation process will ask you to select which users or roles you wish to permit to administer this key. This is a very sensitive action, and must be constrained to as small of a group as possible. I suggest adding only your most privileged users from your organization.

Step 3 of CMK creation, select the key administrator by searching IAM user or role and selecting the user or role

The users of this key must also be defined. This must also be as limited of a group as possible, but it will need to include those whom you will allow to read your session log data as well as the IAM role that you use with your instance profile. For example, if you are using a custom role “EC2RoleSSMCWSecureLogs” as the IAM role for your EC2 instances, then this role will need key usage permissions and will need to be selected in the list.

Step 4 of CMK creation, search for EC2RoleSSMCWSecureLogs IAM role and select the roles to have permission to use the key

Choose Next, and choose Finish to complete the CMK creation. You will see a success message that the customer master key was created with the alias “session-manager-logs” (or the name that you had specified in Step 2).

Next, we must manually edit the key policy to include a section that grants the CloudWatch Logs service access to this key. Select the alias or Key ID hyperlink to view the details of the key that we just created. Note the Amazon Resource Name (ARN) for the newly created key, as we will need this value when creating the secure log group. Under the Key Policy section, select the “Switch to policy view” button.

Highlighted in the box, ARN for the key under general configuration section and edit button for key policy

In the resulting screen, select the “Edit” button to manually edit the key policy.

Paste the policy statement section given as follows into the policy as a new grant, being sure to replace <REGION>, <ACCOUNTNUMBER>, and <LOGGROUPNAME> with the appropriate values.

{ "Sid": "Allow access for CloudWatch secure log group", "Effect": "Allow", "Principal": { "Service": "logs.<REGION>.amazonaws.com" }, "Action": [ "kms:Encrypt*", "kms:Decrypt*", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:Describe*" ], "Resource": "*", "Condition": { "ArnEquals": { "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:<REGION>>:<ACCCOUNTNUMBER>:log-group:<LOGGROUPNAME>" } } }

Screen showing key policy in edit mode and section to allow access for CloudWatch secure log group named “ssm-secure-consolelog-group” added

Select Save Changes to save the policy changes made. Note the ARN for the key. With this step completed, we now have created a Customer Managed symmetric key and given the log group named “ssm-secure-consolelog-group” permission to use the key for the actions specified in the action block of the policy.

Step 2: Create CloudWatch Logs log group

For the next step, we will create an encrypted log group with the same name that we had specified in the policy section for the CMK.

From the CloudWatch Logs console, select Create log group.

Create log group screen under CloudWatch with “Create log group” button highlighted

In the Log group details page, make sure that you enter the same name value that was specified in the policy for the CMK (“ssm-secure-consolelog-group” in our case). Select a retention period that matches your need, and enter the ARN for the new KMS key that we noted earlier. Select the create button to complete the creation of the Log group.

Specifying the log group name as ssm-secure-consolelog-group, retention period of 14 days, and KMS key ARN with create button highlighted

With the encrypted log group created, next we will configure our Amazon EC2 instances to send session manager console access logs to the secure CloudWatch log group that we created.

Step 3: Update your EC2 instance profile

For your EC2 instances to be able to send the Session Manger logs to CloudWatch, you must update the IAM Role used by your EC2 instance by permitting it to communicate with the CloudWatch Logs log group that you just created. The AmazonSSMManagedInstanceCore managed policy doesn’t include these permissions, so they must be added manually prior to using Session Manager.

As part of the prerequisites, you should have already set up a role attached with an AWS Managed AmazonSSMManagedInstanceCore policy and used it as the IAM Role for your EC2 Instances. In the following example, you can see an IAM role “EC2RoleSSMCWSecureLogs” that is used by the EC2 instance.

EC2 instance details screen with IAM Role section having “EC2RoleSSMCWSecureLogs” highlighted

Selecting the IAM Role hyperlink will open Amazon IAM with the role selected. You must add the following policy to your existing role (“EC2RoleSSMCWSecureLogs” in this example), which will allow the local SSM agent to use your new log destination, again being sure to replace the <REGION>, <ACCOUNTNUMBER> and <LOGGROUPNAME>.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "logs:DescribeLogGroups", "Resource": "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:<LOGGROUPNAME>:log-stream:*", "arn:aws:logs:<REGION>:<ACCOUNTNUMBER>:log-group:<LOGGROUPNAME>" ] } ]
}

You will add this inline policy-start by selecting the Add inline policy link. This will open the policy editor.

IAM Role details page of “EC2RoleSSMCWSecureLogs” with “Add inline policy” link highlighted

In the policy editor, select the JSON tab, and then copy and paste the policy statement. Again, remember to replace the ,<REGION>, <ACCOUNTNUMBER> and <LOGGROUPNAME> with values that are accurate for you. Make sure that there are no errors, and select the Review Policy button.

 Step 1 of create inline policy with policy text added to the editor and review policy button highlighted

In the Review policy page, specify a name for the inline policy statement, and select the Create policy button.

Step 2 of creating inline policy with inline policy name specified and create policy button highlighted

The policy is now created inside the Role as an inline policy.

Step 4: Enable logging to Amazon CloudWatch Logs

Now that we have properly configured the KMS keys, key access policy, CloudWatch log group, and access for EC2 to the CloudWatch log group, we can enable the delivery of session logs to CloudWatch. To do so, navigate to the AWS SSM console, select Session Manager, then Preferences, and you can see that the KMS Encryption and CloudWatch logging are disabled by default. You must enable it for the solution to work. Select the Edit button to proceed.

Screen showing AWS SSM Session Manager preference with KMS encryption and CloudWatch logging as disabled. The edit button is highlighted

First, enable KMS encryption by selecting the check-box and selecting the KMS key with the name that we created earlier from the drop down of KMS Keys.

AWS SSM Session Manager preference screen in edit mode with KMS encryption enabled highlighted and KMS key alias selected from the drop down menu

Scroll down the preference page to the section on CloudWatch. Here, enable CloudWatch logging, and then select the log group that you created earlier.

AWS SSM Session Manager preference screen in edit mode with CloudWatch logging option enabled and ssm-secure-consolelog-group selected from the CloudWatch log groups

Save your changes to complete the process. This concludes the set-up part of the solution. Now, to check if we have the solution working, start a session using the session manager, and type a few commands in the session prompt. Open CloudWatch and under Logs menu, select Log groups and select the secure log group that we created, and we can see log files with information about the session, the commands that were entered, and the response that was displayed to the user during the session.

CloudWatch ssm-secure-consolelog-group view showing session manager logs. Session data section of log showing command executed on EC2 is highlighted

Troubleshooting

Most often, when issues prevent logs from being uploaded to CloudWatch Logs, you can diagnose these from the Amazon SSM Agent logs. If you don’t see logs being delivered to CloudWatch as expected, then check the logs from your EC2 instance (or on-premises server), and see if errors are related to key permissions. Here is an example of a misconfigured key permission mapping:

grep ERROR /var/log/amazon/ssm/amazon-ssm-agent.log |grep ssm-session-worker | tail 2021-03-29 23:09:29 ERROR [ssm-session-worker] [[email protected]] [DataBackend] [pluginName=Standard_Stream] error when calling AWS APIs. error details - AccessDeniedException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/managed-ec2-ssm-role/i-024259ed24a2cbaeb is not authorized to perform: logs:DescribeLogGroups on resource: arn:aws:logs:us-east-1:<ACCOUNT_NUMBER>:log-group::log-stream:
2021-03-29 23:09:29 ERROR [ssm-session-worker] [[email protected]] [DataBackend] [pluginName=Standard_Stream] Validation failed, err: We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.

Conclusion

Part 1 of this series outlines the process to save and search AWS SSM Session Manager logs in Amazon CloudWatch. We created an encryption key using Amazon KMS and used the key to encrypt Session Manager logs that are sent to Amazon CloudWatch. In Part 2, we will show you how to search through the logs using the CloudWatch console, and configure a metric filter based on a filter pattern in a new metric namespace. We will also configure the log data to be sent to Amazon OpenSearch for detailed analytics.

About the authors

Rich McDonough

Rich McDonough is a Sr. WW CloudOps Specialist Solutions Architect for AWS based in Toronto. His primary focus is Cloud Operations, helping customers scale their use of AWS safely and securely, and guiding customers in their adoption of observability practices and services. Before joining AWS in 2018, he specialized in helping migrate customers into the cloud.

Manoj Subhadevan

Manoj Subhadevan is an Engineering Leader and Software Development Manger with Amazon Mobile Shopping Foundation. His team is responsible for release engineering of Amazon Mobile Shopping App and uses AWS technologies to achieve the scale. During his free time, he listens to music and plans to travel the world.