Upgrading Terraform EKS Module to Version 20

It is essential to keep your Terraform modules up to date to leverage new features, enhancements, and security improvements. Recently, the Terraform EKS module released version 20, which serves as a transition to the upcoming version 21. This version introduces a few breaking changes, including the deprecation of certain keys and the introduction of a submodule for managing the aws-auth ConfigMap. Upgrading to version 20 now prepares your infrastructure for a smoother transition to version 21, which will completely remove the aws-auth.

Prerequisites

Before starting the module migration, review the official upgrade documentation. It is crucial to test your code locally using terraform init and terraform plan, and make sure only apply changes that matches your code. In our case, we manage our Terraform workspace remotely on Terraform Cloud, so the guide below references Terraform Cloud as our Terraform CI/CD pipeline.

Backwards Incompatible Changes

Minimum supported AWS provider version increased to v5.34. Minimum supported Terraform version increased to v1.3 to support advanced features like Terraform state moved blocks. The resolve_conflicts argument within the cluster_addons configuration has been replaced with resolve_conflicts_on_create and resolve_conflicts_on_update. The default value for the preserve argument of cluster_addons is now set to true. The Karpenter sub-module's use of the IRSA naming convention has been removed, and the Karpenter controller IAM policy has been updated. The aws-auth ConfigMap resources have been moved to a standalone sub-module. Support for cluster access management has been added with the default authentication mode set as API_AND_CONFIG_MAP. Karpenter EventBridge rule key spot_interrupt updated to correct the misspelling (was spot_interupt).

Upcoming Changes in v21.0

The aws-auth sub-module will be removed entirely. The default value for authentication_mode will change to API. Possible replacement of the launch template and autoscaling group usage within the EKS managed nodegroup and self-managed nodegroup sub-modules with the terraform-aws-autoscaling module. The platform variable will be replaced by ami_type to better associate the correct user data format to the respective OS.

Migration Process

1. Upgrade EKS Module Version and AWS Provider a. Update EKS Module Version Update the EKS module version in your Terraform configuration file.

-   version = "~> 19.0"
+   version = "~> 20.0"

b. Comment Out Deprecated Keys Comment out the deprecated keys related to manage_aws_auth_configmap, aws_auth_roles, and aws_auth_users.

# manage_aws_auth_configmap = true

# aws_auth_roles = [
#   {
#     rolearn  = module.eks_admin.iam_role_arn
#     username = module.eks_admin.iam_role_name
#     groups   = ["system:masters"]
#   },
#   {
#     rolearn  = data.terraform_remote_state.openid_connect.outputs.github_oidc_eks_staging_arn
#     username = data.terraform_remote_state.openid_connect.outputs.github_oidc_eks_staging_name
#     groups   = ["system:masters"]
#   },
# ]

# aws_auth_users = [
#   {
#     userarn  = module.eks_user.iam_user_arn
#     username = module.eks_user.iam_user_name
#     groups   = ["system:masters"]
#   }
# ]

c. Upgrade AWS Provider Update the AWS provider version to ensure compatibility with the new EKS module.

-      version = "~> 4.0"
+      version = "~> 5.34"

d. Initialize Terraform Run the following command to ensure the provider and module are initialized successfully.

$ terraform init -upgrade

2. Use aws-auth Submodule Implement the aws-auth submodule to rebuild the aws-auth ConfigMap. Ensure all node groups are correctly mapped.

module "eks_aws_auth" {
  source  = "terraform-aws-modules/eks/aws//modules/aws-auth"
  version = "~> 20.0"

  manage_aws_auth_configmap = true

  aws_auth_roles = [
    {
      rolearn  = module.eks_admin.iam_role_arn
      username = module.eks_admin.iam_role_name
      groups   = ["system:masters"]
    },
    {
      rolearn  = data.terraform_remote_state.openid_connect.outputs.github_oidc_eks_staging_arn
      username = data.terraform_remote_state.openid_connect.outputs.github_oidc_eks_staging_name
      groups   = ["system:masters"]
    },
    {
      rolearn  = module.eks.eks_managed_node_groups.default_node_group.iam_role_arn
      username = "system:node:{{EC2PrivateDNSName}}"
      groups   = ["system:bootstrappers", "system:nodes"]
    },
    // Add other node groups here
  ]

  aws_auth_users = [
    {
      userarn  = module.eks_user.iam_user_arn
      username = module.eks_user.iam_user_name
      groups   = ["system:masters"]
    }
  ]
}

3. Add New Authentication Mode Update the EKS module configuration to include the new authentication_mode default value.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"
  
  // Other configurations
  authentication_mode = "API_AND_CONFIG_MAP"
  // Other configurations
}

4. Validate and Plan Run terraform plan to ensure all changes align correctly with your configuration.

5. Commit Changes and Review Create a new branch, commit your changes, and let Terraform Cloud run a plan remotely. Review the changes to ensure they are as expected.

6. Update Authentication Mode in AWS Console In the AWS EKS console, update the authentication mode to ConfigMap -> EKS API and ConfigMap. Save the changes and wait for the cluster status to return to Active.

7. Finalize and Apply Changes Discard the previous Terraform plan in Terraform Cloud, start a new plan, review the changes, and apply them. Ensure the resources module.eks_aws_auth.kubernetes_config_map_v1_data.aws_auth[0] and module.eks.kubernetes_config_map_v1_data.aws_auth[0] are correctly updated.

8. Verify Log into your EKS cluster and run a few commands to verify that everything is functioning correctly.

Conclusion

Upgrading Terraform AWS EKS Module to version 20 sets you up for future changes in version 21, making it easier to manage cluster access and enhance overall security. It's a key step to keeping your cloud setup up-to-date and being ready for what's next.