Horde: the road to v1.0

By: Derek Kraan / 2019-01-17

I have been working hard these last couple of weeks on Horde and DeltaCrdt. I already hinted at the changes that are coming on Twitter, but now I’d like to write a little more in-depth about the changes.

If you are wondering what Horde and DeltaCrdt are, you can read the introduction blog posts (Horde, DeltaCrdt).

If you are using Horde, please fill out this survey. It will help me a lot!

Changes in Horde 0.3.0

The biggest focus of Horde 0.3.0 has been API parity with Elixir’s own Registry. This should help anyone trying to transition to multi-node with an existing project. Horde’s Registry is almost a drop-in replacement for Elixir’s Registry. This means that, for example, it is now mandatory to specify keys: :unique in Horde.Registry.start_link/1. I have also deprecated Horde.Registry.processes/1, as it does not match the API (and there is an alternative available in Horde.Registry.match(registry, :_)).

Horde.Registry will now also clean up the registry when a process in the registry exits. This comes with a very large caveat: in order to ensure that Horde.Registry doesn’t self-implode, it is no longer possible to rejoin two different instances of Horde.Registry after a network partition event (Horde simply refuses to do so). This is because we have no way of knowing whether a particular process has actually gone down or if the node it was living on has disconnected. The way to restore your cluster is to restart the instance of Horde.Registry and rejoin it to the larger part of the cluster.

Processes are now also linked to Horde.Registry when they are registered. This means that if Horde.Registry dies (or is killed, after a network partition), the process registered in the registry will receive an exit signal. You can let your process die, in which case it will be restarted by the supervisor and reregistered with the registry, but if you do not want this, then you should implement the correct handle_info callback to ensure that the process reregisters itself with Horde.Registry.

The changes are large, but they also get us really close to being able to offer Horde.Registry as a drop-in replacement for Elixir.Registry.

Changes in DeltaCrdt 0.3.0

DeltaCrdt has gained one very important feature. DeltaCrdt now implements join decompositions to reduce back-propagation. In previous versions of DeltaCrdt, it was possible for deltas to be propagated unnecessarily, which increases overhead, degrading overall system performance. Join decomposition helps us crack this problem by allowing DeltaCrdt to split a delta into many small deltas (in fact, each delta must be indivisible: it may only contain one mutation). Then we can evaluate per mini-delta whether it has already been incorporated into the state. If it has, then we have seen it already (and have already queued it for transmission to neighbours) and will ignore it.

This extra computation comes with a cost, of course, but it should make the performance of DeltaCrdt much more predictable, especially when dealing with multiple nodes (>2).

The path to a 1.0 release

With these changes, we have made an important step towards a 1.0 release for both Horde and DeltaCrdt. However, there are still a number of features that I would like to include before I feel confident enough to release a 1.0:

Drop us a line

Get the ball rolling on your new project, fill out the form below and we'll be in touch quickly.

Recent Posts

Walkman - isolate your tests from the world

By: Derek Kraan / 2019-07-22

Introducing MerkleMap: improving Horde's performance

By: Derek Kraan / 2019-05-20

What's new in Horde v0.5.0

By: Derek Kraan / 2019-05-06

Why should every process be supervised?

By: Derek Kraan / 2019-04-01

Implementing Connection Draining in Phoenix

By: Derek Kraan / 2019-01-24

Avoid these OTP Supervision performance pitfalls

By: Derek Kraan / 2019-01-17