Suraj Kumar

Problem

Vulnerability posture was known only from periodic scan reports that were stale the day they landed. Nobody could answer "is that CVE running in production right now, and where" without manual cross-referencing between a scanner and the cluster.

Engineering decisions

1. Correlate against running workloads, not registries

   Scanning images in a registry tells you what might be vulnerable. Joining scan results against what is actually scheduled in the cluster tells you what is exposed — a far shorter and more actionable list.

2. Prioritise by exposure, not raw CVSS

   A critical CVE in an internal batch job outranks nothing. Severity is weighted by whether the workload is internet-facing, so remediation effort follows real risk.

3. Built for the auditor as well as the engineer

   The same data backs both the engineering triage view and the evidence a banking regulator asks for, so compliance reporting stopped being a separate manual exercise.

Problem

One platform team supported several client engagements at once. Routine requests — namespaces, scaling, config, inspecting a failing workload — arrived as tickets and chat messages, making the platform team the bottleneck for everyone else's delivery.

Engineering decisions

1. An API, not a wiki page

   Documentation asks people to do the work correctly; an endpoint does it correctly by construction. FastAPI gave typed request validation and generated docs, so the interface explained itself.

2. One abstraction over three clouds

   Teams targeted EKS, AKS and GKE. Exposing one consistent resource model meant developers never had to learn which cloud their cluster happened to run on.

3. Guardrails inside the endpoint

   Quotas, naming and permission boundaries are enforced by the service rather than by review. Self-service without governance is just a faster way to create drift.

Problem

Nomad and Puppet failures were discovered reactively — usually when something downstream broke. There was no continuous signal on whether the orchestration and configuration layer was actually healthy across the fleet.

Engineering decisions

1. Poll the services that everything else depends on

   Monitoring applications while ignoring the scheduler and config-management layer leaves the most consequential failures invisible. The monitor targets the dependencies, not just the dependents.

2. Alert into the existing pipeline

   A new dashboard nobody watches is not monitoring. Output went into the alerting system engineers already responded to, so detection turned directly into action.

3. Paired with cost work

   The same fleet visibility that surfaced failures also exposed idle and oversized resources, which fed the cloud cost optimisation effort.

Problem

Connecting a growing device fleet securely is the hard part of IoT. Manual per-device provisioning does not scale, and a shared credential baked into firmware becomes a single catastrophic key.

Engineering decisions

1. Device Provisioning Service over manual registration

   DPS lets each device attest and receive its own identity on first boot. Onboarding scales with the fleet instead of with operator time, and there is no shared secret to leak.

2. Separate ingestion from processing

   IoT Hub absorbs device traffic; backend services on AKS scale independently behind it. A spike in device chatter does not force the application tier to scale in lockstep.

3. Microservices on AKS for availability

   Backend processing was decomposed and deployed for high availability, so a single failing consumer degrades one function rather than the whole platform.

Problem

Databases provisioned inside the cluster had no consistent, verifiable backup path. In a regulated environment that is simultaneously a recovery risk and an audit finding.

Engineering decisions

1. Scheduled in-cluster dumps as CronJobs

   Running the backup where the database lives avoids brittle external access paths and keeps the job lifecycle managed by the same system that manages the workload.

2. S3 lifecycle policies as the retention control

   Retention is expressed as infrastructure configuration rather than as a documented human process, so the control cannot silently drift out of compliance.

3. Recovery objectives drove the schedule

   Backup frequency was derived from the acceptable recovery point, not chosen by convention — the schedule is an answer to a stated RPO.

Problem

Logs created by developers across the company lived in scattered places with no shared record and no way to be notified about your own entries.

Engineering decisions

1. RabbitMQ between ingestion and notification

   Queueing the notification work meant a slow or failing mail path could never block or lose a log write — the two concerns fail independently.

2. Per-developer email digests

   Notification is scoped to the logs you created, so the system stays useful instead of becoming noise everyone filters away.