Eliminate Complexity and Improve Efficiency with an Automated Build and Deployment Workflow

‍

Problem Statement

The Clappia platform is built using Microservices architecture. We have more than 30 backend services, as well as multiple user-facing and internal web and mobile applications, which makes managing build and deployment a complex and time-consuming task. 

To streamline this process and avoid delays in production releases, we have implemented an automated system that manages the code from the moment it is pushed to the repository all the way through the build and test stages, and finally to the production environment, ensuring that all of our applications are consistently and efficiently delivered to our users.

‍

In this article, we describe the setup of this Automated System. 

‍

Multiple AWS Accounts

The Automated Build System uses three AWS Accounts - 

1. Dev account where all the code is stored, build and release procedures are defined. 
2. Staging account where all the resources of our Staging environment are deployed - AWS Lambda, API Gateway, CloudFront, DynamoDB and ElasticSearch Databases.
3. Production account where all the same resources of the Production environment are hosted.

Release Workflow

In this section, we have described the release Workflow.‍

Step 1: Code Push to AWS CodeCommit

The code for the entire platform is stored in AWS CodeCommit. Once the developers commit their changes, these changes are reviewed by other team members and, if approved, merged into the master branch in CodeCommit. This process occurs continually throughout the sprint and may involve the implementation of new features, platform improvements, or bug fixes.‍

Step 2: Scheduled Code Build for Staging Environment

We have configured an AWS EventBridge Rule to automatically trigger the build of our staging environment on a weekly basis at a specific day and time. At this time, the recent changes from CodeCommit are retrieved by CodeBuild. Each repository includes a buildspec.yml file that outlines the build procedures for Staging and Production environments. CodeBuild executes these procedures from the Staging buildspec.yml file and finishes the build process.

Step 3: Release to the Staging Environment

The CodeBuild process defined in the previous step also includes commands for releasing the code to the staging environment. For backend microservices, we use the Serverless framework to deploy to AWS Lambda and API Gateway. Web applications are deployed to AWS CloudFront, and Android apps are pushed to the Internal testing track on the Google Play Store.

Step 4: Testing on Staging Environment

Once all the code moves to the Staging environment, all the changes are tested. Some tests are automated and some are manual. If some issues are found in testing, thenext steps are manually canceled.

Step 5: Scheduled CodeBuild for Production Environment

The code remains in the staging environment for two days, during which it is tested. After this period, another EventBridge Rule triggers AWS CodeBuild to perform a production build. This process also retrieves the recent changes from CodeCommit and executes the production build procedure defined in the production buildspec.yml file.

Step 6: Release to Production

Once the Production CodeBuild finishes, all the production resources - backend services on AWS Lambda and API Gateway, web applications on S3 and CloudFront and mobile Apps on Google PlayStore are also released.

This completes the build and release workflow.

The same is repeated automatically on a Weekly basis.

Benefits of the Automated System

Following are some of the benefits that we got after setting up the Automated Release System:

1. Zero DevOps effort: The Automated system can handle the entire build and deployment process without the need for manual intervention from DevOps teams. This saves time and effort and allows the team to focus on the core business requirements.‍
2. More frequent releases: The Automated system handles multiple builds and deployments in a short amount of time, allowing for more frequent releases of updates and new features. This has allowed us to have production releases every week.‍
3. Clarity to customers and business teams on timelines: We are able to provide clear and predictable timelines for releases, which can be shared with customers and business teams. This allows for better planning and coordination and helps to set realistic expectations.‍
4. Less chances of errors: The Automated system has reduced the potential for human error, which leads to fewer errors and a more reliable platform overall.‍
5. Expanding the release process to other regions: This process can easily be replicated when setting up new resources in other regions. We simply need to create copies of the CodeBuild projects and buildspec.yml procedures, and the system will handle the release process for the new regions as well.‍
6. Secure access: Since we have split the entire process across multiple AWS Accounts, it is easier to manage access of the resources. For example, the developers have access only to the Dev and Staging accounts, the testers have access only to the Staging environment and there is no need to give access of the Production account to any individual unless there is a critical issue.