Monitoring and alerting on Terraform Cloud Workspaces for any misconfiguration

Monitoring and alerting on Terraform Cloud Workspaces for any misconfiguration

As engineers tackling tasks together from different time zones, we have to keep ourselves aligned when we pass work on from one person to the next. Part of this is dividing our team into three major timezones: APAC, EU, and Americas, where we handle all of our communication online and with standup meetings placed at times that allow us to sync between timezones. However because of these fragmented work schedules and async communication, sometimes configuration falls out of sync. This is how we solve it.

The Problem

Not not far back, infrastructure was manually created using what has become known as “ClickOps” referring to the manual selection of configurations and settings using the clicks of a mouse, which wasn’t very automation-friendly and was prone to human error.

Nowadays, “Infrastructure-as-code” allows infrastructure to be configured and set up with all the components defined in code. The beauty of this approach is that it introduces useful practices from software developers, like, for instance, the use of Pull Requests (peer reviewing) where other engineers double-check any changes before approving it worthy of being deployed, reducing the chances of human error.

Infrastructure-as-code in Terraform allows the creation of components (like servers, databases, networks) with “plans” or “manifests” detailing what is needed in the desired infrastructure setup. Services like Terraform Cloud, which we use on a daily basis, automatically sync with GitHub to manage infrastructure changes on each code push event. Changes in code on GitHub trigger Terraform Cloud to plan and offer a review of these changes before they are actually applied, bringing it to the attention of the infra developers, further enhancing control, and reducing errors.

This is all great, however, we’ve been observing issues with Terraform Workspaces reconfigured by an engineer for testing and left in this temporary state. Sometimes incorrectly linked to GitHub or the previous run was left in a planned/error state or not configured to follow the correct code branch. This could be because the engineer is working on a feature branch or because they forgot to switch it back once they were finished working on it. We've observed a growing trend of toil tasks that require us to rectify these discrepancies, which is diverting our time away from productive, billable tasks assigned from our customers.

The Solution

To address this, we decided to build and implement an automated monitoring system written as a Lambda function in Python. The code lives on GitHub and executed with AWS Lambda. In this case we opted for the serverless platform to make it the most cost-effective for this kind of task as it only incurs charges for the time it runs, which, in this case, is not long at all. We decided to run the Lambda function daily at 8 AM GMT to suit the APAC afternoon and the EU morning.

The Lambda function dynamically pulls a list of all Terraform Cloud workspaces and checks each for proper configuration. It assesses GitHub connectivity, branch alignment, and the latest build state and then compiles it all into a summary that’s automatically posted to a private Discord channel on our server. It also highlights any discrepancies to give the team a status overview and allows them to quickly click and address them if needed.

How the solution currently looks

Conclusion

This automated system has significantly improved our efficiency in the infrastructure development process and gave us a peace of mind that all the workspaces will be in the desired state when we need them. It ensures that workspaces are consistently and correctly configured, reducing the time engineers spend on fixing configuration issues. This allows the team to focus more on the development tasks, enhancing productivity and project turnaround times.