Distributed systems and Microservices pt.3

Before we explore some ways to implement microservices, we should first review some questions that need to be answered as early as possible.  If you are more familiar with large “monolithic” applications, moving to microservices can be easy at first, but without proper planning, can cause considerable frustration later on.

Q1: What technology or technologies are you going to use or need?


From a technology perspective nginx has a good series out that can start describing the pieces you will need to consider

Q2: What changes will you need to make to your software design process?

Moving towards a more Domain Driven Design process can also help you find the proper boundaries for your services.

Q3: Can your company support microservices?

You may need to examine your business processes.
Microservices may require you to embrace more of a DevOps or NoOps culture.

Distributed systems and microservices pt.2

What is a microservice?

Many have given thought to formalize a definition for microservices.  Martin Fowler and James Lewis have their opinion, and their article is very close to my own opinion. I would encourage anyone to read through it before continuing here.

I believe many already have formulated their own mental model, and I don’t think I need to completely rehash the past few years.

However, my opinion is that a microservice is; a unit of functionality, that fulfills a need and can reliably deploy and live in isolation.

This doesn’t really speak to size though, and my microservice might be my macroservice or monolith.  I also concede that “unit of functionality” is vague.  If we are talking DDD concepts, we might be talking about one to few bounded contexts.  Existing applications might find it easier to progress towards microservices at a feature level.

Plan to change things

Ultimately you will need to plan for refactoring the boundaries of your services.  My experience has shown, that like many things, you will often encounter the need to remodel boundaries well into your project.

Grok asynchronous actions

“Understanding”of asynchronous actions will not be enough.
Microservices typically lead to distributed services.
Distributed services will often require asynchronous patterns to be applied.

If your background is mostly client/server applications built around request/response communications, this might be very difficult at first.

However, keep in mind that the real world is asynchronous.

Plan for failure early

Even if your system is a simple Proof of Concept, you should familiarize yourself with the fallacies of distributed computing.  Failure happens, and if your are working on a highly distributed system, failures will happen often.  Networks go down, disks die, 3rd party services go down in the middle of the day.

The earlier you at least acknowledge these exist, the better off you will be.


Understand that small tutorials are not production examples

Isolation is not easy

Having a service that can be deployed and live in isolation starts off very easy.  Once you have multiple services, and perhaps multiple versions of those services, you will find dependencies and dependencies will make isolation more difficult.

Integration is not easy

Standing up REST based web services is very simple, regardless of server/client technology.  Moving from explicit request/response patterns into more and more asynchronous patterns will complicate your design.

Furthermore, planning for failure here is tricky, you must be able to plan for failure early on, and once you have services calling services, you need to be building in retry logic and logging and monitoring capable of alerting you to failures.


Time marches on, complexity goes up


Eventually one of your services will go down, so you will need to plan for that..
Do you turn to retry logic?  How many times? How often?

Do you need to inform the user that the request eventually succeeded?
What do you do while you wait for the service to respond?  What if the user has moved on, or left the application entirely?

How do you coordinate long running operations across multiple services?

Can you deal with partial success? Can you roll back to a known state after failure?  Without distributed transactions?


These are the issues we will start tackling next, as we move towards some example implementations.

The real world is asynchronous

I love how simple the world works.

On the drive into work, traffic is always moving perfectly.
When I arrive at my desk, I have all my previous days tasks completed, and all of today’s tasks prepared.
Everytime I send someone an email, I receive an instant response and it always contains the information I needed.
Meetings start on time and everyone invited shows up prepared.

I doubt any person in the world could actually say their days were like that, but many times this is how we design software.
We expect that everything will always run in a reliable and calculated way.

I have seen very skilled and talented developers hit a mental roadblock when they move from synchronous systems to asynchronous ones.

Keep in mind that the real world forces us to cope with the unexpected every day.  You can not safeguard your systems from all failures, delays or other unexpected errors.  You can however put plans in place to cope with these events.

Do not ignore the complexities of asynchronous systems, deal with them head on.

As a benefit, your systems will model explicitly these coping and reconciliation scenarios.


Stream Entity Framework results from WebAPI

Generally we try and keep our web services slim, passing back and forth as little data as possible.

However, there are times where we need to send back large amounts of data directly from a database.

If you try and pull all the data into memory before sending it, you can quickly run into trouble.  DB server load, web server load, and memory can all be exhausted very easily.

The answer is to stream the data back to the client.

If you are using WebAPI and Entity Framework, here is a small example of how to achieve this using async methods:

using System.Data.Entity;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Web;
using System.Web.Http;
using Newtonsoft.Json;

namespace StreamTest.Controllers
    public class DataController : ApiController
        private readonly Data_EDM db = new Data_EDM();

        public HttpResponseMessage GetDataStream(int count)
            HttpResponseMessage response = Request.CreateResponse();

            response.Content = new PushStreamContent(
                async (outputStream, httpContent, transportContext) =>
                        await db.ImportData.AsNoTracking().Take(count).ForEachAsync(async (entity) =>
                            var jsonString = JsonConvert.SerializeObject(entity);
                            var buffer = Encoding.UTF8.GetBytes(jsonString);

                            await outputStream.WriteAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
                    catch (HttpException ex)
                        if (ex.ErrorCode == -2147023667) // The remote host closed the connection. 
                        // Close output stream as we are done
            return response;

Distributed systems and Microservices

I’ve been working on cloud hosted, event based, distributed systems for a few years now.

For the next few weeks and months, I’ll be writing about all the benefits and trade-offs I’ve encountered, the various technologies I’ve used, and things that came up later in the development process.

Anything you find here should be viewed as one persons account of their experiences, what you will find if you go down this road is that consensus doesn’t exist, and there are far too many choices to begin to declare any tool, library or process as “the best way”.