larlet-fr-david-cache/cache/2024/2c0b2588dfcd3a194da4133c7505cd3e/index.md

title: A comparison of JavaScript CRDTs
url: https://blog.notmyidea.org/a-comparison-of-javascript-crdts.html
hash_url: 2c0b2588dfcd3a194da4133c7505cd3e
archive_date: 2024-03-25
og_image: https://blog.notmyidea.org/images/umap/crdt-converge.png
description: 
favicon: https://blog.notmyidea.org/favicon-32x32.png
language: fr_FR

<p>Collaboration is one of the most requested features on <a href="https://umap-project.org">uMap</a>. 
I’ve talked <a href="https://blog.notmyidea.org/tag/umap.html">in previous articles</a> how
we could add real-time features “the simple way”, by:</p>
<ul>
<li>a) catching when changes are done on the interface ;</li>
<li>b) sending messages to the other parties and ;</li>
<li>c) applying the changes on the receiving client.</li>
</ul>
<p>This works well in general, but it doesn’t take care of conflicts handling, especially when a disconnect can happen.</p>
<p>For this reason, I got more into “Conflict-free Resolution Data Types” (CRDTs), with the goal of understanding what they are, how they work, what are the different libraries out there, and which one would be a good fit for us, if any.</p>
<p>As things are changing quickly in this field, note that this article was written in March 2024.</p>
<hr>
<div class="toc">
<ul>
<li><a href="#part-1-what-are-crdts">Part 1 - What are CRDTs?</a><ul>
<li><a href="#why-using-crdts">Why using&nbsp;CRDTs?</a></li>
<li><a href="#traditional-data-synchronization-methods">Traditional data synchronization&nbsp;methods</a></li>
<li><a href="#solving-complex-cases">Solving complex&nbsp;cases</a></li>
<li><a href="#last-write-wins-registers">Last Write Wins&nbsp;Registers</a></li>
<li><a href="#state-based-vs-operation-based">State-based vs Operation&nbsp;based</a></li>
<li><a href="#how-the-server-fits-in-the-picture">How the server fits in the&nbsp;picture</a></li>
<li><a href="#how-offline-is-handled">How offline is&nbsp;handled</a></li>
</ul>
</li>
<li><a href="#part-2-javascript-crdts">Part 2: JavaScript CRDTs</a><ul>
<li><a href="#the-demo-application">The demo&nbsp;application</a></li>
<li><a href="#yjs">Y.js</a></li>
<li><a href="#automerge">Automerge</a></li>
<li><a href="#json-joy"><span class="caps">JSON</span>&nbsp;Joy</a></li>
</ul>
</li>
<li><a href="#part-3-comparison-table">Part 3: Comparison table</a><ul>
<li><a href="#working-with-patches">Working with&nbsp;patches</a></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
</li>
<li><a href="#extra-notes">Extra notes</a><ul>
<li><a href="#yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></a></li>
<li><a href="#resources">Resources</a></li>
</ul>
</li>
</ul>
</div>
<hr>
<h2 id="part-1-what-are-crdts">Part 1 - What are CRDTs?</h2>
<p>Conflict-free Resolution Data Types are a family of data types able to merge their states with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.</p>
<p>CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.</p>
<p><span class="dquo">“</span>Append-only sets” are probably one of the most common type of <span class="caps">CRDT</span>: you can add the same element again and again, it will only be present once in the set. It’s our old friend <code>Set</code>, as we can find in many programming languages.</p>
<h3 id="why-using-crdts">Why using CRDTs?</h3>
<p>For uMap, CRDTs offer a solution to several challenges:</p>
<ol>
<li>
<p><strong>Simultaneous Editing</strong>: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other’s contributions. We need all the replicas to converge to the same state.</p>
</li>
<li>
<p><strong>Network Latency and Partition</strong>: uMap operates over networks that can experience delays or temporary outages (think editing on the ground). CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.</p>
</li>
<li>
<p><strong>Simplified Conflict Resolution</strong>: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.</p>
</li>
<li>
<p><strong>Server load</strong>: uMap currently relies on central servers (one per instance). Adopting CRDTs could help lower the work done on the server, increasing resilience and scalability.</p>
</li>
</ol>
<h3 id="traditional-data-synchronization-methods">Traditional data synchronization methods</h3>
<p>Traditional data synchronization methods typically rely on a central source of truth (the server) to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.</p>
<p>In contrast, CRDTs leverage mathematical properties (the fact that the data types can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.</p>
<p>This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.</p>
<h3 id="solving-complex-cases">Solving complex cases</h3>
<p>At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures.</p>
<p>Fortunately for us, our use case is comparatively straightforward, and we probably only need <span class="caps">LWW</span> registers.</p>
<h3 id="last-write-wins-registers">Last Write Wins Registers</h3>
<p>As you might have guessed from the name, a <span class="caps">LWW</span> register is a specific type of <span class="caps">CRDT</span> which “just” replaces the value with the last write. The main concern is establishing the sequence of updates, to order them together (who is the last one?).</p>
<p>In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.</p>
<p>To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.</p>
<blockquote>
<p>A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.</p>
<p>– <a href="https://en.wikipedia.org/wiki/Vector_clock">Wikipedia</a></p>
</blockquote>


<p>Note that we could also use a library such as <a href="https://github.com/pubkey/rxdb">rxdb</a> — to handle the syncing, offline, etc. — because we have a master: we use the server, and we can use it to handle the merge conflicts.
But by doing so, we would give more responsibility to the server, whereas when using CRDTs it’s possible to do the merge only on the clients (enabling no-master replications).</p>
<h3 id="state-based-vs-operation-based">State-based vs Operation based</h3>
<p>While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. It turns out most of the CRDTs implementation I looked at are operation-based, and propose an <span class="caps">API</span> to interact with them as you’re changing the state, so <strong>it doesn’t really matter</strong> in practice.</p>
<blockquote>
<p>The two alternatives are theoretically equivalent, as each can emulate the
other. However, there are practical differences. State-based CRDTs are
often simpler to design and to implement; their only requirement from the
communication substrate is some kind of gossip protocol. <strong>Their drawback is that
the entire state of every <span class="caps">CRDT</span> must be transmitted eventually to every other
replica, which may be costly</strong>. In contrast, operation-based CRDTs transmit only
the update operations, which are typically small.</p>
<p><a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">Wikipedia on CRDTs</a></p>
</blockquote>
<h3 id="how-the-server-fits-in-the-picture">How the server fits in the picture</h3>
<p>While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronization would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers. It makes it easy for the server to send back the needed information to the clients (for new peers, or if the peers didn’t cache the data locally).</p>
<p>In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be “brainless” and just relay the messages.</p>
<p>For automerge, it would mean the provider will “just” handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)</p>
<h3 id="how-offline-is-handled">How offline is handled</h3>
<p>I was curious about how offline editing might work, and what would happen when going back online. Changes can happen both online and offline, making no difference for the “reconciliation” step. When going back online, a “patch” is computed by the newly reconnected peer, and sent to the other peers.</p>
<hr>
<h2 id="part-2-javascript-crdts">Part 2: JavaScript CRDTs</h2>
<p>Now that we’re familiar with CRDTs and how they can help us, let’s create a map application which syncs marker positions, on different browsers.</p>
<p>We’ll be comparing three JavaScript libraries: <a href="https://yjs.dev/">Y.js</a>, <a href="https://automerge.org/">Automerge</a> and <a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a>, considering:</p>
<ol>
<li><strong>Their external <span class="caps">API</span></strong>: is it easy to use in our case? What are the challenging parts?</li>
<li><strong>Community and Support</strong>: What is the size and activity of the developer community / ecosystem?</li>
<li><strong>Size of the <span class="caps">JS</span> library</strong>, because we want to limit the impact on our users browsers.</li>
<li><strong>Efficiency</strong>: Probe the bandwidth when doing edits. What’s being transmitted over the wire? </li>
</ol>
<p>I setup a demo application for each of the libraries. Everything is available <a href="https://gitlab.com/umap-project/leaflet-sync">in a git repository</a> if you want to try it out yourself.</p>
<h3 id="the-demo-application">The demo application</h3>
<p>All the demos are made against the same set of features. It</p>
<ul>
<li>Creates a marker when the map is clicked</li>
<li>Moves the markers on hover.</li>
</ul>
<p>This should probably be enough for us to try out.</p>
<p>Here’s the whole code for this, using <a href="https://leafletjs.com/">Leaflet - a JavaScript library for interactive maps</a>. </p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nx">L</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"leaflet"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="s2">"leaflet/dist/leaflet.css"</span><span class="p">;</span>

<span class="c1">// Create a map with a default tilelayer.</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="s2">"map"</span><span class="p">).</span><span class="nx">setView</span><span class="p">([</span><span class="mf">48.1173</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="mf">1.6778</span><span class="p">],</span><span class="w"> </span><span class="mf">13</span><span class="p">);</span>
<span class="nx">L</span><span class="p">.</span><span class="nx">tileLayer</span><span class="p">(</span><span class="s2">"https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">maxZoom</span><span class="o">:</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span>
<span class="w">    </span><span class="nx">attribution</span><span class="o">:</span><span class="w"> </span><span class="s2">"© OpenStreetMap contributors"</span><span class="p">,</span>
<span class="p">}).</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span>

<span class="c1">// Features contains a reference to the marker objects, mapped by the uuids</span>
<span class="kd">let</span><span class="w"> </span><span class="nx">features</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span>

<span class="c1">// An upsert function, creating a marker at the passed latlng.</span>
<span class="c1">// If an uuid is provided, it changes the coordinates at the given address</span>
<span class="kd">function</span><span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">uuid</span><span class="p">)</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">crypto</span><span class="p">.</span><span class="nx">randomUUID</span><span class="p">();</span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span><span class="w"> </span>

<span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">features</span><span class="p">).</span><span class="nx">includes</span><span class="p">(</span><span class="nx">uuid</span><span class="p">))</span><span class="w"> </span><span class="p">{</span>
<span class="w">        </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">];</span>
<span class="w">        </span><span class="nx">marker</span><span class="p">.</span><span class="nx">setLatLng</span><span class="p">(</span><span class="nx">latlng</span><span class="p">);</span>
<span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span>
<span class="w">        </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">marker</span><span class="p">(</span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w">            </span><span class="nx">draggable</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w">            </span><span class="nx">autoPan</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w">            </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">uuid</span><span class="p">,</span>
<span class="w">        </span><span class="p">});</span>

<span class="w">        </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span>
<span class="w">        </span><span class="nx">marker</span><span class="p">.</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span>
<span class="w">        </span><span class="nx">marker</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">target</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">            </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">);</span>
<span class="w">        </span><span class="p">});</span>
<span class="w">    </span><span class="p">}</span>
<span class="p">}</span>

<span class="c1">// Add new features to the map with a click</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">});</span>
<span class="p">});</span>
</code></pre></div>

<p>It does the following:</p>
<ul>
<li>Creates a map zoomed on Rennes, France</li>
<li>Maintains a <code>features</code> object, referencing the added markers</li>
<li>Provides a <code>upsertMarker</code> function, creating a marker on the map at the given latitude and longitude, and updating its latitude and longitude if it already exists.</li>
<li>It listens to the <code>click</code> event on the map, calling <code>upsertMarker</code> with the appropriate arguments.</li>
</ul>
<p>Note that the data is not “reactive” (in the sense of React apps): there is no central state that’s updated and triggers the rendering of the user interface.</p>
<h3 id="yjs">Y.js</h3>
<p>Y.js is the first library I’ve looked at, probably because it’s the oldest one, and the more commonly referred to.</p>
<p>The <span class="caps">API</span> seem to offer what we look for, and provides a way to <a href="https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent">observe changes</a>. Here’s how it works:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="kr">as</span><span class="w"> </span><span class="nx">Y</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"yjs"</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"y-websocket"</span><span class="p">;</span>

<span class="c1">// Instanciate a document</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">doc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Y</span><span class="p">.</span><span class="nx">Doc</span><span class="p">();</span>
</code></pre></div>

<p>When we add a new marker, we update the <span class="caps">CRDT</span> (<code>markers.set</code>).</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">markers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">.</span><span class="nx">getMap</span><span class="p">(</span><span class="s2">"markers"</span><span class="p">);</span>
<span class="nx">markers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">options</span><span class="p">.</span><span class="nx">uuid</span><span class="p">,</span><span class="w"> </span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span>
</code></pre></div>

<p>Another connected peer can observe the changes, like this:</p>
<div class="highlight"><pre><span></span><code><span class="nx">markers</span><span class="p">.</span><span class="nx">observe</span><span class="p">((</span><span class="nx">event</span><span class="p">,</span><span class="w"> </span><span class="nx">transaction</span><span class="p">)</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">  </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">transaction</span><span class="p">.</span><span class="nx">local</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">event</span><span class="p">.</span><span class="nx">changes</span><span class="p">.</span><span class="nx">keys</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">change</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">)</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">      </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">markers</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">);</span>
<span class="w">      </span><span class="k">switch</span><span class="p">(</span><span class="nx">change</span><span class="p">.</span><span class="nx">action</span><span class="p">){</span>
<span class="w">          </span><span class="k">case</span><span class="w"> </span><span class="s1">'add'</span><span class="o">:</span>
<span class="w">          </span><span class="k">case</span><span class="w"> </span><span class="s1">'update'</span><span class="o">:</span>
<span class="w">              </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span>
<span class="w">              </span><span class="k">break</span><span class="p">;</span>
<span class="w">          </span><span class="k">case</span><span class="w"> </span><span class="s1">'delete'</span><span class="o">:</span>
<span class="w">              </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Property "</span><span class="si">${</span><span class="nx">key</span><span class="si">}</span><span class="sb">" was deleted. ".`</span><span class="p">);</span>
<span class="w">      </span><span class="p">}</span>
<span class="w">    </span><span class="p">});</span>
<span class="w">  </span><span class="p">}</span>
<span class="p">});</span>
</code></pre></div>

<p>We cycle on the received changes, and then apply them on our map. In the case of an offline peer coming back online after some time, the <code>observe</code> event will be called only once. </p>
<p>I’m not dealing with the case of marker deletions here, but deleted items are also taken into account. The data isn’t really deleted in this case, but a “tombstone” is used, making it possible to resolve conflicts (for instance, if one people deleted a marker and some other updated it during the same time).</p>
<p>Y.js comes with multiple “connection providers”, which make it possible to sync with different protocols (there is even <a href="https://github.com/yousefED/matrix-crdt">a way to sync over the matrix protocol</a> 😇).</p>
<p>More usefully for us, there is <a href="https://github.com/yjs/y-websocket">an implemented protocol for WebSockets</a>. Here is how to use one of these providers:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// Sync clients with the y-websocket provider</span>
<span class="kd">const</span><span class="w"> </span><span class="nx">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="p">(</span>
<span class="w">    </span><span class="s2">"ws://localhost:1234"</span><span class="p">,</span>
<span class="w">    </span><span class="s2">"leaflet-sync"</span><span class="p">,</span>
<span class="w">    </span><span class="nx">doc</span>
<span class="p">);</span>
</code></pre></div>

<p>This code setups a WebSocket connection with a server that will maintain a local copy of the <span class="caps">CRDT</span>, as explained above.</p>
<p>It’s also possible to send “awareness” information (some state you don’t want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers, if they’re connected, etc.</p>
<div class="highlight"><pre><span></span><code><span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"mousemove"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">  </span><span class="nx">awareness</span><span class="p">.</span><span class="nx">setLocalStateField</span><span class="p">(</span><span class="s2">"user"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">cursor</span><span class="o">:</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span>
<span class="w">  </span><span class="p">});</span>
<span class="p">});</span><span class="w">  </span>
</code></pre></div>

<p>I made <a href="https://gitlab.com/umap-project/leaflet-sync/-/tree/yjs">a quick proof of concept with Y.js</a> in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.</p>
<h4 id="python-bindings">Python bindings</h4>
<p>Y.js has been ported to rust with <a href="https://github.com/y-crdt/y-crdt">the Y.rs project</a>, making it possible to have binding in other languages, like ruby and python (see <a href="https://github.com/y-crdt/ypy">Y.py</a>). The python bindings are currently looking for a maintainer.</p>
<h4 id="library-size">Library size</h4>
<ul>
<li>Y.js is 4,16 Ko</li>
<li>Y-Websocket is 21,14 Ko</li>
</ul>
<h4 id="the-data-being-transmitted">The data being transmitted</h4>
<p>In a scenario where all clients connect to a central server, which handles the <span class="caps">CRDT</span> locally and then transmits back to other parties, adding 20 points on one client, then 20 points in another generates ~5 Ko of data (so approximately 16 bytes per edit).</p>
<p>Pros: </p>
<ul>
<li>The <span class="caps">API</span> was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate.</li>
<li>It seems to be widely used, and the community seems active.</li>
<li>It is <a href="https://docs.yjs.dev/">well documented</a></li>
<li>There is awareness support</li>
</ul>
<p>Cons:</p>
<ul>
<li>It doesn’t seem to work well <a href="https://github.com/yjs/yjs/issues/325">without a <span class="caps">JS</span> bundler</a> which could be a problem for us.</li>
<li>The Websocket connection provider doesn’t do what I was expecting it to, as it requires the server to run a <span class="caps">CRDT</span> locally.</li>
</ul>
<hr>
<h3 id="automerge">Automerge</h3>
<p><a href="https://automerge.org/">Automerge</a> is another <span class="caps">CRDT</span> library started by the folks at <a href="https://www.inkandswitch.com/">Ink <span class="amp">&amp;</span> Switch</a> with Martin Kleppmann. Automerge is actually the low level interface. There is a higher-level interface named <a href="https://automerge.org/docs/repositories/">Automerge-repo</a>. Here is how to use it:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">Repo</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@automerge/automerge-repo"</span><span class="p">;</span>

<span class="kd">const</span><span class="w"> </span><span class="nx">repo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Repo</span><span class="p">();</span>
<span class="kd">let</span><span class="w"> </span><span class="nx">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">repo</span><span class="p">.</span><span class="nx">create</span><span class="p">()</span>
<span class="c1">// or repo.find(name)</span>
</code></pre></div>

<p>To change the document, call <code>handle.change</code> and pass it a function that will make changes to the document.</p>
<p>Here, when a new marker is added:</p>
<div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">change</span><span class="p">((</span><span class="nx">doc</span><span class="p">)</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">doc</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">cleanLatLng</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div>

<p>I had to use a <code>cleanLatLng</code> function in order to not pass the whole object, otherwise it wouldn’t serialize. It’s really just a simple helper taking the properties of interest for us (and letting away all the rest).</p>
<p>Another peer can observe the changes, like this:</p>
<div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"change"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">doc</span><span class="p">,</span><span class="w"> </span><span class="nx">patches</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">  </span><span class="nx">patches</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(({</span><span class="w"> </span><span class="nx">action</span><span class="p">,</span><span class="w"> </span><span class="nx">path</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mf">2</span><span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"insert"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w">      </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">[</span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">]];</span>
<span class="w">      </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">],</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span>
<span class="w">    </span><span class="p">}</span>
<span class="w">  </span><span class="p">});</span>
<span class="p">});</span>
</code></pre></div>

<p>The “patch” interface is a bit less verbose than the one from Y.js. It wasn’t well documented, so I had to play around with the messages to understand exactly what the possible actions were. In the end, it gets me what I’m looking for, the changes that occurred to my data.</p>
<p>Here I’m using <code>path.length == 2 &amp;&amp; action === 'insert'</code> to detect that a change occurred to the marker. Since we don’t make a difference between the creation of a marker and its update (when being moved), it works well for us.</p>
<h4 id="python-bindings_1">Python bindings</h4>
<p>There are <a href="https://github.com/automerge/automerge-py">python bindings</a> for automerge “core”. (It doesn’t — yet — provide ways to interact with “repos”).</p>
<h4 id="library-size_1">Library size</h4>
<ul>
<li>Size of the automerge + automerge repo: 1,64 mb</li>
<li>Size of the WebSocket provider: 0,10 mb </li>
</ul>
<p>This is quite a large bundle size, and the team behind automerge is aware of it and working on a solution.</p>
<h4 id="the-data-being-transmitted_1">The data being transmitted</h4>
<p>In the same scenario, adding 20 points on one client, then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.</p>
<p>Pros:</p>
<ul>
<li>There is an <span class="caps">API</span> to <a href="https://automerge.org/docs/documents/conflicts/">get informed when a conflict occurred</a></li>
<li>Python bindings are currently being worked on, soon to reach a stable version</li>
<li>The team was overall very responsive and trying to help.</li>
</ul>
<p>Cons:</p>
<ul>
<li>The JavaScript is currently generated via Web Assembly, which could make it harder to debug.</li>
<li>The large bundle size of the generated files.</li>
</ul>
<hr>
<h3 id="json-joy"><span class="caps">JSON</span> Joy</h3>
<p><a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a> is the latest to the party.</p>
<p>It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would leave us with the hands free in order to implement the protocol that would work for us.</p>
<p>Here is how to use it. On the different peers you start with different forks of the same document:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="nx">Model</span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s1">'json-joy/es2020/json-crdt'</span><span class="p">;</span>
<span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">s</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch"</span><span class="p">;</span>

<span class="c1">// Initiate a model with a custom ID</span>

<span class="kd">const</span><span class="w"> </span><span class="nx">rootModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Model</span><span class="p">.</span><span class="nx">withLogicalClock</span><span class="p">(</span><span class="mf">11111111</span><span class="p">);</span>

<span class="c1">// populate it with default data</span>
<span class="nx">rootModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">root</span><span class="p">({</span>
<span class="w">    </span><span class="nx">markers</span><span class="o">:</span><span class="w"> </span><span class="p">{},</span>
<span class="p">});</span>

<span class="c1">// Fork it on each client</span>
<span class="kd">let</span><span class="w"> </span><span class="nx">userModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">rootModel</span><span class="p">.</span><span class="nx">fork</span><span class="p">();</span>
</code></pre></div>

<p>When adding a new marker, we can define a new constant, by using <code>s.con</code>…</p>
<div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">obj</span><span class="p">([</span><span class="s2">"markers"</span><span class="p">]).</span><span class="nx">set</span><span class="p">({</span>
<span class="w">    </span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="o">:</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">con</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">),</span>
<span class="p">});</span>
</code></pre></div>

<p>… and then create a patch and send it to the other peers:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">encode</span><span class="p">,</span><span class="w"> </span><span class="nx">decode</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch/codec/verbose"</span><span class="p">;</span>

<span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">flush</span><span class="p">();</span>
<span class="kd">let</span><span class="w"> </span><span class="nx">payload</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">encode</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span>
</code></pre></div>

<p>On the other peers, when we receive a patch message, decode it and apply it:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">decode</span><span class="p">(</span><span class="nx">payload</span><span class="p">);</span>
<span class="nx">model</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span>
</code></pre></div>

<p>We can observe the changes this way:</p>
<div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">onPatch</span><span class="p">.</span><span class="nx">listen</span><span class="p">((</span><span class="nx">patch</span><span class="p">)</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">    </span><span class="nx">patch</span><span class="p">.</span><span class="nx">ops</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">op</span><span class="p">)</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="p">{</span>
<span class="w">        </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">op</span><span class="p">.</span><span class="nx">name</span><span class="p">()</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"ins_obj"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">op</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="mf">0</span><span class="p">][</span><span class="mf">0</span><span class="p">];</span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">view</span><span class="p">().</span><span class="nx">markers</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span>
<span class="w">            </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span>
<span class="w">        </span><span class="p">}</span>
<span class="w">    </span><span class="p">});</span>
<span class="p">});</span>
</code></pre></div>

<p>Similarly to what we did with automerge, we’re having a look into the patch, and filter on the operations of interest for us (<code>ins_obj</code>). The names of the operations aren’t clearly specified by the spec.</p>
<p>Metrics:</p>
<ul>
<li>Size: 143 ko</li>
<li>Data transmitted for 2 peers and 40 edits: (35 bytes per edit)</li>
</ul>
<p>Pros:</p>
<ul>
<li>Small atomic libraries, making it easy to use only the parts we need.</li>
<li>The interface <a href="https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types">proposes to store different type of data</a> (constants, values, arrays, etc.)</li>
<li>Distributed as different type of <span class="caps">JS</span> bundles (modules, wasm, etc.)</li>
<li>Low level, so you know what you’re doing</li>
</ul>
<p>Cons:</p>
<ul>
<li>It doesn’t provide a high level interface for sync</li>
<li>It’s currently a one-person project, without clear community channels to gather with other interested folks.</li>
<li>Quite recent, so probably rough spots are to be found</li>
</ul>
<hr>
<h2 id="part-3-comparison-table">Part 3: Comparison table</h2>
<p>I found Y.js and Automerge quite similar for my use case, while <span class="caps">JSON</span> Joy was taking a different (less “all-included”) approach. Here is a summary table to help read the differences I found.</p>
<table>
<thead>
<tr>
<th></th>
<th>Y.js</th>
<th>Automerge</th>
<th><span class="caps">JSON</span> Joy</th>
</tr>
</thead>
<tbody>
<tr>
<td>Python bindings</td>
<td><a href="https://github.com/y-crdt/ypy">Yes</a></td>
<td><a href="https://github.com/automerge/automerge-py">Yes</a></td>
<td>No</td>
</tr>
<tr>
<td>Syncing</td>
<td>Native <span class="caps">JS</span> structures</td>
<td>Transactional functions</td>
<td>Specific types (bonus points for handling constants)</td>
</tr>
<tr>
<td>Coded in</td>
<td>JavaScript / Rust</td>
<td>TypeScript / Rust</td>
<td>Typescript</td>
</tr>
<tr>
<td>Awareness protocol</td>
<td>Yes, with presence</td>
<td>Yes, without presence</td>
<td>No</td>
</tr>
<tr>
<td>Conflict-detection <span class="caps">API</span></td>
<td>No</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td>Library size</td>
<td><strong>24.3Ko</strong> §<br></td>
<td><strong>1,74 mb</strong> §</td>
<td><strong>143 ko</strong></td>
</tr>
</tbody>
</table>
<p>§ size of the connectors included</p>
<h3 id="working-with-patches">Working with patches</h3>
<p>In order to observe the changes, we need to inspect the given patches and work on what we find. I found the different libraries expose different sets of APIs. All of these APIs were quite a bit hard to find, and it’s not clear if they are public or private.</p>
<p>One thing to keep in mind is that these “patch” events happen only once per patch received. You can see it as a “diff” of the state between the current and the incoming states.</p>
<ul>
<li>Y.js exposes a utility which is able to tell you what the action on the key is (“delete”, “update” and “add”)</li>
<li>Automerge and <span class="caps">JSON</span> Joy, on the other hand, don’t provide such utility functions, meaning you would need to find that out yourself.</li>
</ul>
<h3 id="conclusion">Conclusion</h3>
<p>The goal here is not to tell which one of these libraries is the best one. They’re all great and have their strenghs. None of them implement the high-level <span class="caps">API</span> I was expecting, where the clients talk with each other and the server just relays messages, but maybe it’s because it’s better in general to have the server have the representation of the data, saving a roundtrip for the clients.</p>
<p>I wasn’t expecting to have a look at patches to understand what changed at the low level. The way it’s currently implemented is very suitable for “reactive” applications, but require more involvement to sync between the CRDTs state and the application state.</p>
<p>In the end, adding CRDTs is made very simple, probably due to the fact all we really need is a sort of distributed key/value store.</p>
<p>I’m not sure yet which library we will end up using for uMap (if any), but my understanding is clearer than it was when I started. I guess that’s what progress looks like 😎</p>

<h3 id="yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></h3>
<p>While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:</p>
<blockquote>
<ul>
<li><strong><span class="caps">RGA</span></strong> [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that’s updated anytime we detect that remote insert has an id
  with sequence number higher that local counter. Therefore, every time we produce
  a new insert operation, we give it a highest counter value known at the time.</li>
<li><strong><span class="caps">YATA</span></strong> [used by Yjs] also uses a single integer value, however unlike in case of <span class="caps">RGA</span> we
  don’t use a single counter shared with other replicas, but rather let each
  peer keep its own, which is incremented monotonically only by that peer. Since
  increments are monotonic, we can also use them to detect missing operations eg.
  updates marked as A:1 and A:3 imply, that there must be another (potentially
  missing) update A:2.</li>
</ul>
</blockquote>
<h3 id="resources">Resources</h3>
<ul>
<li><a href="https://www.youtube.com/watch?v=x7drE24geUw">CRDTs: The Hard Parts</a>, a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren’t solved yet.</li>
<li><a href="https://jakelazaroff.com/words/an-interactive-intro-to-crdts/">An Interactive Intro to CRDTs</a> gets you trough different steps to understand what are CRDTs, and how to implement a <span class="caps">LWW</span> Register.</li>
<li><a href="https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/">Bartosz Sypytkowski</a> introduction on CRDTs, with practical examples is very intuitive.</li>
<li><a href="https://jzhao.xyz/thoughts/CRDT-Implementations"><span class="caps">CRDT</span> Implementations</a> are notes by Jacky which were useful to me when understanding CRDTs.</li>
</ul>