How to ensure consistency in microservices

H

Stick to one write operation at a time.

In a cloud environment, such as Microsoft Azure, you cannot rely on the concept of distributed transactions, as services are not participating.

Furthermore SDK's are often asynchronous in nature and typically have some form of retry built in, causing individual operations to get out of order or occur more than once.

Writing to more than one service at a time, will cause an exponential increase in persistence complexity. Especially when the consumer code also tries to retry the command.

1 Write operation has 3 result states: Success, Failure or Uncertainty (in case of asynchronicity or timeouts)
2 Write operations will potentially have up to 9 combined result states
3 write operations combine up to 27 potential states

To ensure data consistency in a distributed system, without leveraging distributed transactions, each command handler should be atomic, retriable and idempotent

Atomicity

To avoid partial completion scenarios, it's best to let each command handler perform only a single write operation at a time.

When multiple write operations need to be performed, perform them one at a time.

Serially if you have to...

But durable fan-out technology such as topics, or event streams, can be used to parallelize the operations.

Retriability

If any write operation should fail, due to e.g. a transient exceptions, the operation must be retried until it succeeds or some treshold is reached.

Don't retry forever though. Put the operation aside till later after a known number of retries.

Potentially rely on human intervention to invoke the final retry (after fixing a bug or so).

Idempotency

In case an operation does get retried, it may have already succeed before,

So ensure that the outcome of the operation remains the same regardless.

About the author

YVES GOELEVEN

I've been a software architect for over 20 years.

My main areas of expertise are large scale distributed systems, progressive web applications, event driven architecture, domain driven design, event sourcing, messaging, and the Microsoft Azure platform.

As I've transitioned into the second half of my career, I made it my personal goal to train the next generation of software architects.

Get in touch

Want to get better at software architecture?

Sign up to my newsletter and get regular advice to improve your architecture skills.

You can unsubscribe at any time by clicking the link in the footer of your emails. I use Mailchimp as my marketing platform. By clicking subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.