// case study
Automating Containerized AWS Lambda Deployments for Data Analytics
A Git-driven workflow connecting infrastructure provisioning, container builds, and analytics deployment across development, staging, and production.
Role & Context
Role
DevOps Engineer Lead
Company
Cleaver-Brooks
Tenure
Aug 2022 – Sep 2023
Cleaver-Brooks is a century-old industrial manufacturer that was building Prometha, a cloud-native IoT platform for its industrial equipment. The Data Analytics team was one of several engineering groups operating within that platform.
Problem
The Data Analytics team developed Lambda workloads that processed data from Amazon Kinesis streams. Delivering those workloads required coordinating several moving parts: application code, Docker images, an ECR image repository, Lambda infrastructure, and environment-specific configuration for development, staging, and production.\n\nThe infrastructure-management burden associated with that coordination was the problem to solve. My goal was to automate the provisioning and deployment work so the analytics team could focus on writing and reviewing analytics code rather than managing cloud infrastructure.
My Contribution
As DevOps Engineer Lead, I designed and built the automated CI/CD solution. The work I owned connected:\n\nThe team's Git review and merge workflow to automated delivery — merging into the appropriate branch triggered the workflow without requiring manual steps from the analytics team.\n\nEnvironment-specific Terraform configuration maintained by DevOps — Terraform variable files supplied per-environment infrastructure settings, keeping that concern separate from application development.\n\nDocker image builds and publication to Amazon ECR — the container image became the deployment unit for the Lambda function.\n\nProvisioning and deployment of AWS Lambda functions using those container images — Terraform and supporting Python scripts together formed the automation layer.\n\nAn AWS-hosted Terraform backend using S3 for remote state storage and DynamoDB for state locking, enabling team-safe infrastructure operations.
Architecture Diagram
Conceptual reconstruction — not an exported company diagram
Deployment Workflow
Code review and environment progression
The Data Analytics team pushed application code through the standard Git workflow. After review and merge into the appropriate branch — Dev, Stage, or Main — the CI/CD workflow ran automatically, progressing the change through development, staging, and production environments.
Environment-specific infrastructure configuration
The workflow consumed Terraform variable files maintained by the DevOps team. These files supplied the configuration each environment required. This separation meant analytics engineers could concentrate on application logic while DevOps owned the infrastructure settings consumed by the automation.
Container build and image storage
The pipeline built the analytics application into a Docker container image and published it to Amazon Elastic Container Registry (ECR). The container image served as the deployment package for the Lambda workload.
Lambda provisioning and deployment
The automation created the AWS Lambda function using the container image stored in ECR. Terraform and supporting Python scripts formed the automation layer, provisioning the function and its associated infrastructure.
Analytics execution at runtime
Once deployed, the Lambda function consumed data from Amazon Kinesis streams and performed the analytics processing implemented by the Data Analytics team. The deployment workflow delivered the code and infrastructure; the runtime workflow processed streaming data.
Terraform state management
Remote state was stored in Amazon S3, with a DynamoDB table providing state locking. I used an AWS-hosted backend for this workflow during my 2022–2023 tenure to avoid introducing Terraform Cloud and its associated per-seat cost for a workflow that did not require Terraform Cloud's additional features.
Engineering Decisions
Integrating deployment into the existing Git workflow
Code review and merges already governed how the analytics team released changes. Making that same action the trigger for automated infrastructure and deployment work meant engineers could use a process they already understood, without a separate deployment tool or manual handoff to DevOps.
Keeping environment configuration under DevOps ownership
Environment-specific Terraform variable files were maintained by the DevOps team, not embedded in the application repository. This separated two concerns that change at different rates: the analytics application code, which the Data Analytics team iterated on independently, and the infrastructure configuration, which required DevOps judgment to manage safely across environments.
Using container images for Lambda workloads
Docker images stored in ECR provided the application packaging and deployment mechanism. Container images give Lambda workloads a consistent build environment and make the deployment artifact inspectable and reproducible independent of the runtime.
AWS-hosted Terraform backend over Terraform Cloud
S3 and DynamoDB provided remote state storage and locking for this workflow. Terraform Cloud was evaluated but introduced per-seat licensing cost without providing capability this specific workflow required. An S3 backend with DynamoDB locking delivered the same team-safe remote state while keeping infrastructure tooling spend proportional to the workflow's actual needs.
Outcomes
- →Enabled the analytics team to focus on application development rather than infrastructure management.
- →Automated the path from reviewed code to a deployed analytics workload across development, staging, and production.
- →Reduced manual infrastructure and deployment effort for both the analytics and DevOps teams.
- →Provided a repeatable, Git-triggered workflow that was consistent across environments.
- →Avoided additional Terraform Cloud tooling costs by using an AWS-hosted backend appropriate to the workflow's scale.
"This project demonstrates how I approach platform engineering: connect infrastructure automation to the development workflow so application teams can deliver their work with less operational overhead. At Cleaver-Brooks, I applied that approach to containerized Lambda analytics workloads, combining Git-based delivery, Terraform configuration, Docker packaging, and AWS services into a repeatable deployment process."