Designing Client Connections with Fault Tolerance
Customers visiting your website are generally unforgiving. If they encounter slow response times, prominent errors, or incorrect results, their confidence quickly erodes.
All cloud-hosted systems are vulnerable to system faults, whether caused by network outages, system errors, or other factors. Thus, architecting for the cloud means building resilience from day one.
Our first line of defense is to design less brittle client connections. One component that addresses this directly is Polly, an open-source library. The name "Polly" alludes to the cliché of parrots mimicking cries.
Let's start with a plain HTTP client call to an external REST API. Our system needs to call another system's API.
In this hypothetical scenario, the API receives our request, but our client doesn't get a response because of connection issues. Clients typically throw an error and report it to the customer. However, if this happens too often, we risk alienating customers.
Instead of reporting failure immediately, we'll try a different approach. If the call fails, we'll have our client retry it. This is exactly where Polly excels. Let's update our flow.
We now have a retry strategy in place. When an HTTP client call fails, Polly waits and then retries the same call until it succeeds. Polly is powerful and flexible.
Polly lets us control multiple factors: how many times to retry before accepting failure, how long to wait before the next retry, etc. Plus, we can add circuit breakers to temporarily stop calls entirely, e.g., until a networking issue clears. We can also add fallback behaviors (Plan B).
This is already much more resilient. Let's add some more real-world context.
Modern REST APIs are generally secured with access tokens, and our HTTP client now needs to retrieve one.
Access tokens are typically valid for a limited time. After they expire, APIs return an "unauthorized" response.
The good news is that Polly can manage the access token retrieval and refresh cycle for us, freeing our HTTP client from the burden.
When Polly encounters networking issues or server exceptions, it retries the calls. When Polly fails because an access token has expired, it gets a new one and retries the API call. Polly is now the central point for managing resilience.
Let's now add one more layer of resilience. This addition may appear somewhat heavy-handed at first, so let me explain the reasoning.
We're going to apply message queuing. We're now applying the same design pattern to our own client system – not the remote system, mind you.
API calls are now queued for execution, and a separate asynchronous process makes the actual call. Call failures occur within the worker process and on a different time frame. Our actual customers are now insulated.
This is enterprise-grade resilience. Thanks to message queuing, Polly handles retries and unauthorized access failures in isolation. That said, we do need to rethink our user experience.
We can't tell our users that their data was saved – because it wasn't. We've placed a request in a message queue and are waiting for our async process to save the records. How to handle this effectively is another topic.
Another pitfall happens when the backend accidentally succeeds at making the same save request more than once. Our process should be idempotent, meaning additional runs of the same request can't create duplicates.
Anyhow, we're now firmly on the path to eventual consistency – a much more mature approach to distributed computing. We now deliver better customer experiences and more resilient system flows.
