Approaching Basic Strategies For Optimizing Distributed System Performance

In this installment, we'll run through an exercise in optimizing a hypothetical distributed system's performance. Our example system is a simple, cloud-hosted API backed by a SQL database.

Our goal is to deliver a system that meets inbound request service-level agreements (SLAs) with acceptable data consistency. This article discusses strategies for intelligently directing efforts toward this.

As an aside, this newsletter previously ran a much coarser, now-retracted take on queueing that cut corners. It loosely presented message queueing as the go-to solution for performance issues.

While technically valid, this lacked precision. Moreover, it obscured how to think about message queues and distributed systems.

We'll cover how to approach queueing. First, though, we'll start with a high-level approach to optimizing our SQL data access performance. Then we'll introduce queueing.

Here's the basic setup: an Add Record request is sent to our API. The API then performs a series of SQL queries within an ACID transaction. In the plainest terms, we're reading from and writing to the database with strong consistency and durability.

If we're doing things right, we've already instrumented the codebase. The resulting telemetry data will be invaluable to us as we troubleshoot. Moreover, it will feed observability into SRE operations as we approach go-live.

Assuming we've deployed to a test environment and have a load-testing tool suite (e.g., JMeter, k6) ready to go, we're ready to start the performance-testing cycle. We'll first target internal service-level objectives (SLOs) before production metrics.

Our testing cycle is a straightforward test-measure-fix feedback loop. We identify API operations failing to meet their SLOs, then find and fix the root cause.

So, let's suppose our Add Record operation is failing to meet SLOs. And, unsurprisingly, telemetry points to data access code as the likely cause. That gives us a starting point for directing optimization efforts.

Data access spans the entire stack. This modified diagram, while still greatly simplified, shows more of the moving parts involved in fulfilling requests.

SQL workloads are generally much more resource-intensive than API code. So, if the telemetry data agrees, we'll likely find the biggest performance gains by starting with our database and queries.

A decent profiler will expose inefficiencies in SQL queries as well as design problems in the database. If we conclusively find and fix our database problems but still can't meet our SLOs, our next stop is the API code.

With a code profiler like JetBrains dotTrace and our telemetry data, we should be able to find and fix any code-level problems. If we exhaust this as an option and still can't meet our SLOs, it's probably time to talk with the infrastructure and database admin teams.

Optimizing networks and databases beyond a certain point requires specialized skills, tools, and training. While generalist full-stack developers should understand the basics, most won't have the same deep expertise as a dedicated professional.

Let's now assume we've tried everything. We've optimized the code and the database. We've partnered with database admins and network engineers to eliminate bottlenecks. We've even tried a distributed memory cache to speed up reads.

However, we still can't get Add Record to meet our SLOs. At this stage, queueing might be a viable solution. Let's modify our system design.

Now, instead of talking directly to the SQL database, the API pushes the Add Record request into a queue. The API responds to the caller acknowledging the request. Then, a background worker picks up the request and writes the records to the database. This impacts the system in several ways.

Foremost, API response times should drop. Writing to queues is generally faster than SQL write transactions. For reference, Azure Service Bus delivers single- to low double-digit millisecond latency with relative ease.

However, this new design is more complex. The background worker is a separate system. We have to account for every edge case of broken communication exchanges (e.g., duplicates). We also need to think through in-between states and the user experience, where appropriate.

Queues are communication channels. Their sole purpose is to route vast numbers of enqueue and dequeue operations reliably. Their speed makes them attractive. But they don't come without a price.

Components carry more responsibility in loosely coupled systems, like this one. Simply copying data-access code from the API isn't enough. Background workers need to validate actions the same way the API does. And they can't violate business rules or disrupt data consistency.

Now for the final trade-off: our system no longer has strong consistency. We've changed the API interface. We need to renegotiate SLOs, which will ultimately impact production metrics – if we're here, we can't avoid the conversation. The previous article didn't address this issue.

We now have eventual consistency, which means faster API responses – but also latency in data becoming consistent. This introduces unique engineering and operational challenges. However, if an organization can adapt, it can be a powerful tool. We just need to think things through before completely embracing it.

One alternative worth considering is using a NoSQL database like Azure Cosmos DB to serve queueing operations. With some effort, a NoSQL database could make pending data available sooner. However, evaluating this as a solution is a separate topic altogether.