What's new in Horde v0.5.0

By: Derek Kraan / 2019-05-06

Horde 0.5.0 has just been released, and there are a bunch of changes, so let’s take a look at some of the highlights!

If you are new to Horde, read Introducing Horde — a distributed Supervisor in Elixir first.

Transient / Temporary processes

Support for :transient and :temporary processes was one of the most requested features, and thanks to the efforts of Ethan Mahintorabi, support was merged in #70.

To accomplish this, he forked Elixir’s DynamicSupervisor. This was necessary to be able to send information back to Horde when the process is finished, so that we can stop tracking it. This has also had the side-effect of simplifying Horde’s internals quite a bit, particularly the graceful shutdown.

One thing to look out for: temporary processes will never be restarted, not even if the node they’re on crashes or is shut down. That doesn’t mean they’re useless, see Why should every process be supervised? for more information.

Performance improvements

Horde should be able to handle tens of thousands of processes without breaking a sweat. Further improvements are also in the pipeline that will allow us to stretch this even further.

Horde.Supervisor.wait_for_quorum/2

A common scenario (if you use Horde’s quorum mechanism) is that the system will be unable to function properly until quorum has been reached, causing a potentially large amount of errors, fouling logs etc. With wait_for_quorum/2, you can delay starting up parts of your system until Horde has quorum and is ready to process requests. For example, you might add a simple GenServer that uses wait_for_quorum/2 in the init/1 callback and put it above Phoenix in your application supervisor.

Horde.Cluster.set_members/2

join_hordes/2 has been replaced by set_members/2. This change was done to lift the responsibility of determining which nodes were in the cluster out of Horde. This allows us to always do the right thing when a netsplit occurs.

set_members/2 is more declarative when compared to join_hordes/2, which means you can never forget to remove a node from your cluster (since you specify the whole list every time).

set_members/2 also makes using Horde with a dynamic number of nodes much easier. Previously, if you wanted to use Horde with Kubernetes with a quorum mechanism, you would have had to have designed your own distribution strategy that compared the connected nodes to the nodes in your Kubernetes cluster. Now, you can simply tell Horde which nodes should be in the cluster, and use one of the standard distribution strategies that Horde provides.

Finally, if you were a fan of join_hordes/2, don’t fret, you can still dynamically grow and shrink the cluster by calling Horde.Cluster.set_members(MyHorde, Enum.map(Node.list, fn n -> {MyHorde, n} end)) when a new node is added to the cluster.

Module-based Supervisor / Registry

Also new in 0.5.0 is module-based Supervisors and Registries. Like Ecto.Repo, this allows you to specify an init/1 callback to configure your supervisor or registry (especially useful for cluster membership) at runtime. This also enables Horde to come up in the correct state after a restart every time.

If your cluster membership is static, then you can also specify the list of members as an option in the child_spec of both Supervisor and Registry.

Improvements to DeltaCrdt

DeltaCrdt has undergone massive changes since the last version of Horde. It should now be much more performant, with constant-time add / update / remove operations. DeltaCrdt also now sends diffs to Horde, instead of requiring Horde to read the entire state of the DeltaCrdt each time and compute its own diffs.

The last details of delta garbage collection has also been worked out in this version, which should mean no memory leakage.

DeltaCrdt also gains the power of Merkle Trees by leveraging the new MerkleMap library (Read about it here)

Finally, DeltaCrdt gained an optional storage behaviour that can be used to persist the CRDT state and recover the last-known state immediately in the event of a crash. This can be useful for applications where network access is extremely limited, but it is not exposed in Horde.

Onwards to 1.0!

Finally, I want to mention that if you are using Horde, to please fill out this very very short survey. It’s valuable to me to know that Horde is being used in production, so that I can gauge when it might be ready for a 1.0 release.

Drop us a line

Get the ball rolling on your new project, fill out the form below and we'll be in touch quickly.

Recent Posts

Where to put startup code in Elixir

By: Derek Kraan / 2019-12-06

Walkman - isolate your tests from the world

By: Derek Kraan / 2019-07-22

Introducing MerkleMap: improving Horde's performance

By: Derek Kraan / 2019-05-20

What's new in Horde v0.5.0

By: Derek Kraan / 2019-05-06

Why should every process be supervised?

By: Derek Kraan / 2019-04-01

Implementing Connection Draining in Phoenix

By: Derek Kraan / 2019-01-24