AWS IoT / Cloud Platform Architecture

Cleaver Brooks — a century-old industrial manufacturer — was launching Prometha, a new SaaS IoT product to connect their industrial equipment (boilers, burners, HVAC systems) to the cloud and give customers real-time operational visibility. The problem was total greenfield: no existing AWS footprint, no DevOps team, no defined architecture.

The challenge was to design and build a production-grade IoT data platform from scratch that could ingest telemetry from industrial edge devices in real time, route it through processing pipelines, feed multiple analytics use cases with different data access patterns, and serve a reliable web application — all with a small team and a mandate to automate everything.

The platform needed to handle real-time MQTT telemetry from industrial edge devices with ordered delivery and replay capability. Three distinct data workloads required three different storage strategies: high-frequency time-series sensor readings, aggregated analytical data, and relational IoT asset data (device registry, customer mappings, configurations).

Beyond ingestion and storage, the requirements included containerized application workloads with managed orchestration, fully automated infrastructure provisioning with no manual steps, a CI/CD pipeline from code commit to production, centralized monitoring with alerting and on-call coverage, third-party security audit readiness, and operational reporting for management visibility into cost and service health.

The platform was designed in four logical layers.

Device Layer: Industrial edge devices publish telemetry over MQTT to AWS IoT Core, which handles device registration, X.509 certificate authentication, and lifecycle management. IoT Core's rules engine routes messages downstream with no custom broker to operate.

Ingestion and Processing: Messages flow from IoT Core into Kinesis Data Streams for ordered, replayable ingestion at scale. Lambda functions consume from Kinesis to transform and fan out data to the appropriate stores. A separate set of Lambdas was built for the data analytics team's ETL jobs, triggered on schedule.

Data Layer: Three purpose-built stores handle distinct access patterns — InfluxDB for time-series sensor readings (high write throughput, native retention policies, downsampling), MongoDB for aggregated analytical data, and MySQL for relational IoT asset data. ElastiCache sits in front of frequently-accessed MySQL queries to reduce latency on read-heavy paths.

Application Layer: The Prometha web application runs in Docker containers on AWS EKS, served behind CloudFront for global edge caching. Container images are stored in ECR. AWS CodePipeline orchestrates the CI/CD flow from commit to production — CodeBuild handles build, test, and Kubernetes deployment via kubectl apply; CodeDeploy is not used, as it does not natively support EKS.

Terraform over CloudFormation: When the project started, both tools were evaluated for infrastructure provisioning. CloudFormation is AWS-native and has no additional tooling overhead, but it couples infrastructure management tightly to AWS APIs and produces verbose templates that are harder to reason about and reuse. Terraform's provider model and HCL syntax produce cleaner, more modular code, and the same tooling extends across every cloud provider and third-party API the platform might need. I recommended Terraform, and it became the standard across the entire platform.

S3 + DynamoDB state backend over Terraform Cloud: Remote state was a requirement from the start — local state is incompatible with team workflows and pipeline runs. Terraform Cloud was considered but introduced a per-user licensing cost that was difficult to justify for a small team running infrequent applies. An S3 bucket for state storage with a DynamoDB table for state locking provided identical team-safe remote state at near-zero marginal cost (a few cents per month in S3 and DynamoDB). This recommendation reduced infrastructure tooling spend significantly without sacrificing any capability the team actually used.

Kinesis over SQS: IoT telemetry arrives in bursts and multiple consumers need the same data — the time-series writer, the analytics Lambda, and a future alerting service. SQS would require fan-out via SNS or duplicate ingestion paths. Kinesis provides native fan-out, ordered delivery within a shard, and a 24-hour replay window. That replay window proved critical during incidents — missed data could be reprocessed without re-ingestion from devices.

Three databases instead of one: Early discussions explored a single general-purpose store, but the data patterns diverged too sharply. Sensor readings arrive thousands per minute and are queried as time-range windows — InfluxDB's columnar compression and native retention policies handle this at a fraction of relational storage cost. Analytical data is queried in aggregate patterns that suit MongoDB's document model. Asset and configuration data is relational and transactional, making MySQL the right fit. The operational overhead of three stores was the deliberate trade-off for query performance and storage efficiency.

EKS over ECS: The team wanted workload portability and more granular scheduling control as the platform matured. EKS added setup complexity but avoided orchestration lock-in and built transferable operational skills that carried forward to later engagements.

Infrastructure was provisioned entirely through IaC with no manual console steps. Terraform managed all AWS resources — VPC, subnets, route tables, NAT gateways, EC2, RDS, ElastiCache, EKS, IoT Core, Kinesis, Lambda, S3, CloudFront, and IAM — organized into reusable modules per layer.

Packer built hardened AMIs with the correct OS baseline, packages, and security configurations baked in, eliminating bootstrap-time configuration drift. Ansible handled post-provisioning configuration management and ran idempotent remediation on any drift detected between deployments. InSpec ran compliance assertions as part of every infrastructure pipeline run to validate that provisioned resources matched the defined security baseline before promotion.

AWS Simple AD was deployed to provide centralized identity and access management across the infrastructure. Windows and Linux instances across every tier were domain-joined, giving engineers and service accounts a single directory for authentication rather than managing per-instance local credentials. This eliminated credential drift across the fleet and simplified user lifecycle management — provisioning or deprovisioning a team member was a single directory operation that propagated across all systems automatically.

Network architecture followed defence-in-depth: public subnets for load balancers only, private subnets for all compute and database tiers, no direct internet egress from application or data layers, and VPC endpoints for AWS service communication to keep traffic off the public internet. CloudFront operated as a global edge layer in front of the load balancers — it runs on AWS edge infrastructure outside the VPC and is not placed in any subnet.

Architecture and network diagrams were produced for both internal documentation and management communication. Troubleshooting runbooks were maintained as living documents alongside the codebase.

On-call operations were established from the start. Grafana dashboards surfaced key pipeline health indicators — Kinesis consumer lag, Lambda error rates and duration, InfluxDB write latency, EKS pod restarts, and RDS connection counts. CloudWatch alarms were tiered: warning thresholds for early-detection notifications, critical thresholds for immediate on-call response.

New DevOps engineers were onboarded through structured sessions covering the full platform topology, runbooks, and incident response procedures. Agile ceremonies — sprint planning, retrospectives, and stand-ups — kept cross-team work coordinated across DevOps, development, data analytics, and QA.

A third-party security firm was engaged to audit the platform against AWS security best practices and CIS benchmarks. Findings were triaged by severity and remediated in prioritized sprints before launch, with all critical and high findings resolved.

IAM policies followed least-privilege throughout — each service used a dedicated role scoped to its minimum required permissions, with no shared credentials or over-permissioned roles. IoT device authentication used X.509 certificates managed through AWS IoT Core's certificate lifecycle APIs. All secrets were stored in AWS Secrets Manager — no credentials in environment variables, configuration files, or code. InSpec compliance checks ran on every infrastructure deployment to catch regressions.

The Lambda automation work spanned two distinct areas. The first was the real-time IoT ingestion pipeline — Lambdas consuming from Kinesis, transforming telemetry payloads, and routing records to InfluxDB, MongoDB, and MySQL. The second was scheduled ETL automation built in collaboration with the data analytics team — Lambdas triggered on cron schedules to aggregate time-series data, generate summary records, and feed the analytical layer without manual intervention.

CodePipeline orchestrated delivery, with CodeBuild handling builds, tests, and Kubernetes deployments through kubectl apply — the CodeBuild IAM execution role was mapped to a Kubernetes RBAC role via the aws-auth ConfigMap, giving the pipeline cluster access without storing credentials. Infrastructure changes followed a separate pipeline — Terraform plan output was reviewed before apply, with InSpec running post-apply to confirm compliance.

Observability was structured at three levels. Infrastructure metrics — CPU, memory, disk, and network — were collected via CloudWatch and surfaced in Grafana alongside application and pipeline metrics. Structured application logs were routed to CloudWatch Logs with log group retention policies to manage cost. Kinesis consumer lag was the primary leading indicator for pipeline health — spikes in lag ahead of errors provided early warning before data loss occurred.

Monthly operational reports were generated covering service health, incident summary, and AWS cost breakdown by service. These were delivered to management alongside architecture documentation to maintain visibility into both operational status and infrastructure spend.

The multi-database approach introduced operational surface area. Three separate backup strategies, three monitoring integrations, and three connection pool configurations added complexity that would not exist with a single store. This was a deliberate trade — sensor data at IoT telemetry volumes made a general-purpose relational database impractical, and the operational investment was bounded by automation.

Kinesis sharding required upfront capacity planning. Shard count determines throughput ceiling, and resharding in production carries downtime risk. Provisioning ahead of expected load avoided a mid-production scaling event at the cost of slightly higher Kinesis spend during early ramp-up.

EKS added meaningful cluster management overhead compared to ECS — control plane costs, node group lifecycle management, and kubeconfig operations. For a small DevOps team, this was a real ongoing cost. It paid off in operational familiarity with Kubernetes patterns that transferred directly to subsequent roles operating Kubernetes in larger environments.

The Prometha IoT platform launched as Cleaver Brooks' first cloud-native SaaS product. The infrastructure was fully automated with zero manual provisioning steps, enabling reliable and repeatable environment creation from a single pipeline run.

The third-party security audit improved the overall security posture, with all critical and high findings remediated before launch. The data analytics team's Lambda automation eliminated manual ETL work, letting the team focus on analysis rather than data wrangling. A new DevOps engineer was successfully onboarded using the documented architecture, runbooks, and training sessions. Management maintained full visibility into cost and service health through automated monthly reporting.

The platform established the cloud engineering foundation for Cleaver Brooks' IoT product line and demonstrated the value of treating infrastructure as code from day one — every environment, every configuration, and every compliance check was reproducible, auditable, and version-controlled.