For customers, governing and auditing IP address allocation for at-scale networks is a complex, time-consuming, and potentially error-prone task. This is especially true with application workloads migrating to the cloud at a rapid pace. Traditionally, network administrators have resorted to using combinations of spreadsheets, home-grown tools, and scripts to track address assignments across multiple accounts, Amazon Virtual Private Clouds (Amazon VPC), and Regions. It takes time and care to avoid errors when updating spreadsheets while application development teams request IP address assignments. Should these errors go unnoticed, they can lead to address conflicts and subsequent downtime, thereby causing serious operational and business issues. Consequently, the time taken to deploy the infrastructure changes, such as VPCs and subnets across multiple accounts in an organization, is increased.

The recently launched IPAM feature lets you easily plan, track, and monitor IP addresses for your AWS workloads. Our solution is an alternative approach to this feature that lets you manage IP addresses across multiple accounts in multiple organizations. Moreover, our solution uses AWS CodePipeline workflow, which is part of AWS Control Tower Customizations.

We will demonstrate how to automate a flexible IP Address Management solution that can be used across your AWS Control Tower multi-account environment. This will let you provision your Amazon VPCs. Furthermore, you can provision VPCs with flexibility in their CIDR sizes and manage your overall IPv4 address scheme in a simple way by leveraging AWS Control Tower Customizations, otherwise known as CfCT.

Solution overview

The AWS CodePipeline workflow configures AWS CodePipeline, AWS CodeBuild projects, and AWS Step Functions to orchestrate AWS CloudFormation management within your organization. A customized configuration package is uploaded to an Amazon Simple Storage Service (Amazon S3) bucket, which triggers this workflow and configures the IPAM solution. An Amazon DynamoDB table is provisioned that stores the IP addresses with flexible CIDR sizes. Then, these are leveraged by AWS Lambda functions for deploying VPCs and their subnets in the spoke accounts. We use AWS Identity and Access Management (IAM) roles and AWS Security Token Service (AWS STS) to set up cross-account access between AWS accounts for the Lambda functions. This setup is shown in figure1 below.

Flexible IP Address Management for AWS Control Tower Account Provisioning

Figure 1 : Flexible IP Address Management for AWS Control Tower Account Provisioning

Example walkthrough

In this section, we consider a Supernetwork (e.g., 10.95.0.0/16) that we use as an input for our IPAM solution. This Supernetwork can be tweaked to accommodate any CIDR range. As an end user, you can choose between the options – small, medium, and large CIDR – to deploy the VPC and its subnets during account creation. The first /18 block is used for VPCs or accounts that require a large number of IP addresses or LargeCIDR. The remaining two /19 blocks are used for accounts and VPCs with MediumCIDR Size and SmallCIDR requirements.

The following is an example translation:

  • Small: 10.95.64.0/19
    • With /26 CIDR Mask for VPC,
      • Supports 128 Accounts
      • 64 available IPs per VPC
  • Medium: 10.95.96.0/19
    • With /24 CIDR Mask for VPC,
      • Supports 32 Accounts
      • 256 available IPs per VPC
  • Large: 10.95.0.0/18
    • With /23 CIDR Mask for VPC,
      • Supports 32 Accounts
      • 512 available IPs per VPC

Deployment solution

Prerequisites

  1. The default accounts and the OU structure provisioned through AWS Control Tower are present. In other words, the following accounts are present and enrolled in the AWS Control Tower landing zone:
    • Management Account
    • Log Archive and Audit Accounts (in Security OU)
  2. Create Shared Services OU and create a Network Hub Account. Associate the Network Hub Account to the Shared Services OU.
  3. Deploy the initial-setup.yml CloudFormation template in the management account. This template creates the SSM parameters and AWS Resource Access Manager (AWS RAM) shares to interact with AWS Organizations.

Deployment instructions

The deployment of this Flexible IPAM solution consists of four steps:

Step 1 – Upload the .zip files to Amazon S3 bucket in the Networking Hub Account.

Step 2 – Make the necessary changes to the Management Account.

Step 3 – Provision a spoke account, and enroll to the required OU (e.g., Development OU).

Step 4 – Validate the VPC creation in the spoke account.

Step 1 – Upload the .zip files to the Amazon S3 bucket in the Network Hub Account

  1. Log in to the Network Hub Account, and navigate to Amazon S3.
  2. Create an Amazon S3 bucket in the Network Hub Account using the template storage-bucket.yml. This bucket is used as a storage location for items to be retrieved by other AWS Control Tower customizations.
  3. Select the Amazon S3 bucket created in the previous step. Create a folder named “Networking”. Upload the ip-populator.zip and lambda-for-vpc-subnets-automation.zip to the “Networking” folder.

Note that the Amazon S3 bucket has a bucket policy with “GetObject” access given for all of the accounts in the specified Organization.

Step 2 – Make necessary changes to the Management Account

  1. Deploy the customizations for AWS Control Tower in the Management Account. The framework can be deployed using the ‘Launch in the AWS Management Console’ button in the link provided. Use the default parameters when deploying the framework.
  2. Navigate to CloudFormation in the console. In the left pane, select stacks. Validate that it is in CREATE_COMPLETE state. This takes approximately 15 minutes to complete.
  3. Download the custom-control-tower-configuration.zip file from the provided link.
  4. Update the manifest.yaml file with values that meet your requirements. Modify the Organizational Unit (OU) in the “deployment targets” section of “vpc-automation-in-spoke-dev” template, and update the “Regions” section with the appropriate Region.
  5. Navigate to the parameters directory and update the parameter values in vpc-cidr-populator.json with the required CIDR ranges for the VPC for small, medium, and large configurations. Update the Amazon S3 bucket name parameter with the Amazon S3 bucket created in the Network Hub account.
  6. Modify the parameter values in vpc-automation-in-spoke.json with the VPC and subnet masks as required. Additionally, update the pTShirtSize parameter with one of the allowed values – small, medium or large.
  7. Update the Amazon S3 bucket parameter value in vpc-automation-in-networking-hub.json with the Amazon S3 bucket created in the Network Hub account.
  8. Update the parameters of the manifest.yaml, compress/zip the files, and rename the compressed file to custom-control-tower-configuration.zip.
  9. Upload the custom-control-tower-configuration.zip to the custom-control-tower-configuration-<accountId>-<cfct-region> Amazon S3 bucket located in the Management Account. Uploading the .zip here will initiate the customizations build and propagate the stack sets to the designated Network Hub Account and Deployment OU described in the manifest.yaml file.
  10. Once the customizations pipeline is finished, navigate to the CodePipeline console in the Management Account to validate that the Custom-Control-Tower-CodePipeline is in the Succeeded state. Navigate to the StackSets section of the CloudFormation console to further validate and see the deployed StackSets.

Step 3 – Provision a Spoke Account and enroll to the required OU (e.g., PrePod OU)

  1. Provision a new member account in your AWS Control Tower Landing zone with Account Factory.
  2. Navigate to the Account Factory page in AWS Control Tower.
  3. Select the Enroll account item near the top of the page, and enroll the account to the required OU.
  4. Navigate back to the Management Account and update the manifest.yaml file with the OU above (in which the new account has been provisioned) in the “deployment targets” section of the “vpc-automation-in-spoke-dev” template. Then, update the “Regions” section with the appropriate Region.
  5. Now repeat Steps 9, 10, and 11 in the Management Account.

Step 4 – Validate the VPC creation in the spoke account

  1. After completing the above 3 steps, log in to the spoke account to validate the VPC creation. The VPCs are created based on the size that you provide (small, medium, or large) and CIDRs in the DynamoDB table present in the Network Hub account.

Integrating IPAM solution across multiple organizations

For integrating IP Address Management solution across multiple organizations, follow the next steps. In this scenario, we consider two organizations – Main Organization and PreProd Organization.

The PreProd Organization acts as a staging environment to make sure that everything functions properly with the newly developed features and configurations.

  1. Make sure that the prerequisites are satisfied for this environment as mentioned above.
  2. Download the custom-control-tower-configuration.zip file from the PreProd Organization folder.
  3. Download the ip-populator.zip from the PreProd Organization folder.
  4. Follow the same deployment instructions as performed above, but this time perform the actions in the management account, Network Hub account, and spoke accounts in the PreProd environment.
  5. Modify the parameters in the vpc-cidr-populator.json and vpc-automation-in-spoke.json, as per the provided instructions in the .json files.
  6. Upload the custom-control-tower-configuration.zip to the custom-control-tower-configuration-<accountId>-<cfct-region> Amazon S3 bucket located in the PreProd Management Account.
  7. Once the customizations pipeline is finished, navigate to the CodePipeline console in the PreProd management account to validate that the Custom-Control-Tower-CodePipeline is in the Succeeded state. Navigate to the StackSets section of the CloudFormation console to further validate and see the deployed StackSets.

How does the IPAM solution integrate with existing accounts and VPCs

To populate the DynamoDB table with all of the existing VPC CIDRs from the existing accounts in a given Region, adhere to the following steps:

  1. Log in to the AWS account in which you have your existing VPCs that you would like to integrate with our solution.
  2. Navigate to the CloudFormation console, and deploy the “automation-for-existing-accounts.yml” template.
  3. Input two parameters – the DynamoDB table name and the Network Hub Account ID. The table name can be found in the AWS Systems Manager Parameter Store in the management account (“org/sharedservice/networking/DynamoDBforVPCCIDR”). Additionally, you can get the table name by navigating to the DynamoDB console in the Network Hub account.
  4. Deploy the template by accepting the prompts.
  5. After the stack has been successfully deployed, you can navigate back to the Network Hub account to see the existing VPC CIDRs populated in the DynamoDB table.
  6. The same steps can be repeated in multiple accounts as required.

Note that if the DynamoDB table already has the CIDR and it conflicts with one of the VPC CIDR’s queries by the Lambda function in the above template, then it logs an information message to the CloudWatch Logs. Furthermore, it doesn’t populate the conflicting CIDRs to the table.

Cleanup

After successful testing and validation, all of the resources deployed through CloudFormation templates should be deleted to avoid any unwanted costs. Simply go to the CloudFormation console, identify the appropriate stacks, and delete them.

Note that if you use a multi-account setup, then you must navigate through account boundaries and follow the above steps as needed.

Conclusion

This solution for IP address management that leverages AWS Control Tower customizations demonstrates how easy it is to centrally manage and size your VPCs as per your application requirements. The solution is integrated end-to-end with CfCT. This provides a DevOps approach for the IPAM solution, which creates an up-to-date overview of the IP address space that is available to your organization. Moreover, provides the flexibility of having different CIDR sizes – small, medium, and large. We hope that you’ve found this post informative, and we look forward to hearing how you use our solution!

Shiva Vaidyanathan

Shiva Vaidyanathan is a Senior Cloud Infrastructure Architect at AWS. He provides technical guidance, design and lead implementation projects to customers ensuring their success on AWS. He works towards making cloud networking simpler for everyone. Prior to joining AWS, he has worked on several NSF funded research initiatives on how to perform secure computing in public cloud infrastructures. He holds a MS in Computer Science from Rutgers University and a MS in Electrical Engineering from New York University.

Mokshith Kumar

Mokshith Kumar is a Cloud Infrastructure Architect at AWS. He thrives on learning new technologies and solving complex customer challenges. He enjoys interacting with customers and strives to help accelerate their cloud adoption journey by providing technical guidance and implementation of AWS Solutions. He holds a master’s degree specializing in Computer Networks from the University of Southern California. Off work, Mokshith is an avid swimmer and enjoys listening to music.