Cloud platforms are usually presented as tidy collections of services with clearly defined responsibilities and predictable behavior. In production, that’s almost never what happens. Real systems are messy, interconnected, and full of dependencies that don’t show up in architecture diagrams until something goes wrong.
After working with enterprise cloud environments for years, I’ve seen outages caused by things nobody initially considered part of the problem. A customer-facing authentication issue might look like a failure in a cloud messaging service, but the actual issue could be somewhere completely different; a telecom provider, a mobile carrier, or even the user’s device. The end result is the same: the code never arrives.
That’s why resilience work has shifted so heavily toward recovery testing and operational readiness. Frameworks like DORA and the UK operational resilience standards are increasingly asking organizations to demonstrate how systems recover under stress, not just whether a service technically meets an uptime target.
Best Practices, Hidden Dependencies and Failure Chains
Reference architectures are written for steady state. They assume predictable traffic, well-behaved clients, and dependencies that respond within their advertised latency. Production rarely cooperates. A canonical “best practice” pattern, such as an autoscaling group behind a managed load balancer with retries enabled at every hop, can amplify failure rather than absorb it the moment a downstream service slows down. Retries become retry storms. Autoscaling behind a saturated database makes recovery harder, not faster.
The CrowdStrike incident of July 2024, which crashed roughly 8.5 million Windows endpoints worldwide, was a global reminder that a “low-risk” routine update, deployed without staged validation, can take down critical infrastructure in hours. Best practices are necessary but not sufficient. What matters is whether it has been tested under the conditions you expect to fail in.
Most large-scale systems depend on more than one cloud service, and often on more than one provider. Consider a customer-facing application that delivers one-time passwords by SMS through a managed messaging service such as Amazon SNS, Azure Communication Services, or a comparable GCP path. The visible dependency is the cloud service itself. The invisible chain extends further: through the telecom carrier the cloud provider has partnered with, the mobile network operator the end user actually subscribes to, and the user’s handset. A temporary outage anywhere along that chain produces the same user-visible symptom, which is that the OTP never arrives. Designing only against the headline service overstates the resilience of the system. Including alternative channels, such as email, authenticator apps, or federated identity through Google or Apple, is not an enhancement. It is the actual design.
The same pattern shows up in contact center platforms, which typically depend on serverless compute for custom logic, a managed database for customer records, and AI services for chat and voice routing. Some availability factors aren’t even within the cloud provider’s control. Telecom interconnects, third-party adapter upgrades (the Salesforce-to-cloud-telephony integration is a classic example), and even browser-level differences in local storage permissions can affect whether a feature works for a given user. Failure chains rarely respect service boundaries.
Messaging Systems and the Complexity Tax
Messaging systems such as Amazon SQS, managed RabbitMQ or ActiveMQ brokers, Azure Service Bus, and GCP Pub/Sub are meant to connect across separated components or microservices. In practice, they are where complexity concentrates. A broker showing slow response times or connection failures often isn’t the cause. It is the symptom. The actual culprit is usually upstream or downstream: a containerized producer caught in a loop and flooding the queue, a consumer that retries aggressively without backoff, or an identity provider, whether an LDAP server, a managed directory, or an on-premises AD, that has become slow to authenticate broker connections.
These systems also carry hidden infrastructure dependencies that are easy to miss. A managed message broker typically runs on the same cloud provider’s underlying compute fabric. That means a regional issue in that compute layer can surface as a broker outage, even though the documentation treats them as independent services. Multi-AZ and multi-region deployments mitigate this, but only if they are designed in from the start.
A practical example: a customer I worked with had a five-minute downtime tolerance and ran a single-instance managed broker on the critical path. The provider’s weekly maintenance window rebooted the broker and produced roughly 25 minutes of unavailability, well outside tolerance. The fix was not application code. It was topology. Migrating to a three-node cluster meant maintenance only ever took one node offline at a time, and patching stopped being a user-visible event. No client libraries, no pipelines, and no release processes had to change. Sometimes the most effective reliability improvement is an infrastructure decision the application team never has to worry about.
Designing Around Real Failure
Many organizations still start with uptime percentages and work backward from there. Someone decides the service needs “99.9% availability,” and then engineering teams are expected to justify the architecture afterward.
In practice, reliability decisions make far more sense when they start with business impact instead. Some outages are inconvenient. Others become exponentially more expensive after only a few minutes. Payment systems, customer communications, and real-time operations tend to escalate very quickly once disruptions begin.
The systems companies actually run are usually far more complicated than the diagrams used to describe them. Designing around real failure behavior rather than ideal conditions ultimately determines whether a platform remains resilient when production environments become unpredictable.
