Consul: Service Discovery in a Microservice World

What is Service Discovery?

Service discovery is the concept of separating how processes and services find and communicate with one another. Typically this involves each service/node announcing themselves, some form of storage of the name and host/port of the service, and the ability to query (discover) this data from other services. In effect and in short, decoupling services.

Don't let "microservice" in the title of the post fool you. Service discovery is important on any form of the SOA continuum, whether you have 2 or 200 services.

Why Service Discovery? What's wrong with current approaches?

There are two most common forms of approaching service reachability.

  1. Directly contacting the host (Diagram 1)
  2. Directly contacting a load balancer or proxy (Diagram 2)

Typical inter-service communication

The first has the obvious disadvantage of being a single point of failure and lacking any form of resilience and is absolutely discouraged in any form.

The second is slightly better. You still directly address a host but your application services sit on nodes that the load balancer will route traffic to. If a node goes down then most load balancers will remove it from the pool and traffic will not be routed to it (hopefully). The load balancer will track the health of the nodes and manage their routing. Importantly, however, case two still suffers from a single point of failure and it still tightly couples Service A to Service B. If you decide to move Service B or change its location you have to update all consuming services (including Service A) with the new address.

What's so special about Consul?

Consul is a service discovery implementation built by the guys at Hashicorp (who also created a lot of other awesome tools including Vagrant and Packer).

The particularly genius part though is how they've implemented it. Most service discovery systems are invasive to an application in that each application must intermittently announce itself to a central cluster to tell it that's it's alive and available for traffic. When it wishes to remove itself from discovery it issues an unannounce command. Services will store a cache of other service definitions and therefore understand how to contact those services. The diagram below shows a typical topology:

Consul takes a different approach by building on top of DNS. In practice this means that each service node in your system contains the Consul agent (a process running on each box). The Consul agent is configured (flat JSON file) to know how to query the status/health of the service. Using the diagram above that may be, as an example "service-b-01.example.com:3000/service-status". If DNS doesn't suit you then it also provides a HTTP API. The agents are also responsible for reporting into the Consul server cluster.

Consul is also highly fault tolerant. If the whole service cluster for Consul goes down then this doesn't stop discovery. It does this by using a Gossip Protocol to manage membership and broadbast to the cluster. This means that Consul isn't just a typical client-to-server system, it's also client-to-client. In addition the server nodes use the Raft concensus algorithm to provide consistency for leader election.

Consul is datacenter aware by design. In fact the DNS schema in consul is addressable by datacenter. Service lookups are of the format:

[tag.]<service>.service[.datacenter][.domain]

So we might do:

$ dig @127.0.0.1 -p 8600 service-a.service.dc1.consul. SRV

Which would return

;; ANSWER SECTION:
service-a.service.dc1.consul.  0   IN  SRV 1 1 3000 service-a-01.node.dc1.consul.

;; ADDITIONAL SECTION:
service-a-01.node.dc1.consul. 0   IN  A   10.1.10.12  

You'll notice above, because it's just a DNS query, I can query directly for SRV records in case the service is running on a non-standard port 80.

Does it have a shiny Web UI?

Of course. You can have a play with the online demo here http://demo.consul.io/ui/

Consul Demo UI

Conclusion

This post really just touches the surface of what Consul is and how it works. Consul Template was recently released which is also incredibly useful and interesting if you're using HAProxy, Nginx or similar.

There's a lot of community tools gaining traction, particularly around Docker with both docker-consul and registrator looking interesting projects in the world of Docker.

Author

Ryan Tomlinson

Director of Engineering, hobby photographer and tech blogger