5 changed files with 1362 additions and 0 deletions
+ 631
- 0
cache/2024/2c0b2588dfcd3a194da4133c7505cd3e/index.html
View File
| @@ -0,0 +1,631 @@ | |||
| <!doctype html><!-- This is a valid HTML5 document. --> | |||
| <!-- Screen readers, SEO, extensions and so on. --> | |||
| <html lang="fr"> | |||
| <!-- Has to be within the first 1024 bytes, hence before the `title` element | |||
| See: https://www.w3.org/TR/2012/CR-html5-20121217/document-metadata.html#charset --> | |||
| <meta charset="utf-8"> | |||
| <!-- Why no `X-UA-Compatible` meta: https://stackoverflow.com/a/6771584 --> | |||
| <!-- The viewport meta is quite crowded and we are responsible for that. | |||
| See: https://codepen.io/tigt/post/meta-viewport-for-2015 --> | |||
| <meta name="viewport" content="width=device-width,initial-scale=1"> | |||
| <!-- Required to make a valid HTML5 document. --> | |||
| <title>A comparison of JavaScript CRDTs (archive) — David Larlet</title> | |||
| <meta name="description" content="Publication mise en cache pour en conserver une trace."> | |||
| <!-- That good ol' feed, subscribe :). --> | |||
| <link rel="alternate" type="application/atom+xml" title="Feed" href="/david/log/"> | |||
| <!-- Generated from https://realfavicongenerator.net/ such a mess. --> | |||
| <link rel="apple-touch-icon" sizes="180x180" href="/static/david/icons2/apple-touch-icon.png"> | |||
| <link rel="icon" type="image/png" sizes="32x32" href="/static/david/icons2/favicon-32x32.png"> | |||
| <link rel="icon" type="image/png" sizes="16x16" href="/static/david/icons2/favicon-16x16.png"> | |||
| <link rel="manifest" href="/static/david/icons2/site.webmanifest"> | |||
| <link rel="mask-icon" href="/static/david/icons2/safari-pinned-tab.svg" color="#07486c"> | |||
| <link rel="shortcut icon" href="/static/david/icons2/favicon.ico"> | |||
| <meta name="msapplication-TileColor" content="#f7f7f7"> | |||
| <meta name="msapplication-config" content="/static/david/icons2/browserconfig.xml"> | |||
| <meta name="theme-color" content="#f7f7f7" media="(prefers-color-scheme: light)"> | |||
| <meta name="theme-color" content="#272727" media="(prefers-color-scheme: dark)"> | |||
| <!-- Is that even respected? Retrospectively? What a shAItshow… | |||
| https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ --> | |||
| <meta name="robots" content="noai, noimageai"> | |||
| <!-- Documented, feel free to shoot an email. --> | |||
| <link rel="stylesheet" href="/static/david/css/style_2021-01-20.css"> | |||
| <!-- See https://www.zachleat.com/web/comprehensive-webfonts/ for the trade-off. --> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <script> | |||
| function toggleTheme(themeName) { | |||
| document.documentElement.classList.toggle( | |||
| 'forced-dark', | |||
| themeName === 'dark' | |||
| ) | |||
| document.documentElement.classList.toggle( | |||
| 'forced-light', | |||
| themeName === 'light' | |||
| ) | |||
| } | |||
| const selectedTheme = localStorage.getItem('theme') | |||
| if (selectedTheme !== 'undefined') { | |||
| toggleTheme(selectedTheme) | |||
| } | |||
| </script> | |||
| <meta name="robots" content="noindex, nofollow"> | |||
| <meta content="origin-when-cross-origin" name="referrer"> | |||
| <!-- Canonical URL for SEO purposes --> | |||
| <link rel="canonical" href="https://blog.notmyidea.org/a-comparison-of-javascript-crdts.html"> | |||
| <body class="remarkdown h1-underline h2-underline h3-underline em-underscore hr-center ul-star pre-tick" data-instant-intensity="viewport-all"> | |||
| <article> | |||
| <header> | |||
| <h1>A comparison of JavaScript CRDTs</h1> | |||
| </header> | |||
| <nav> | |||
| <p class="center"> | |||
| <a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use> | |||
| </svg> Accueil</a> • | |||
| <a href="https://blog.notmyidea.org/a-comparison-of-javascript-crdts.html" title="Lien vers le contenu original">Source originale</a> | |||
| <br> | |||
| Mis en cache le 2024-03-25 | |||
| </p> | |||
| </nav> | |||
| <hr> | |||
| <p>Collaboration is one of the most requested features on <a href="https://umap-project.org">uMap</a>. | |||
| I’ve talked <a href="https://blog.notmyidea.org/tag/umap.html">in previous articles</a> how | |||
| we could add real-time features “the simple way”, by:</p> | |||
| <ul> | |||
| <li>a) catching when changes are done on the interface ;</li> | |||
| <li>b) sending messages to the other parties and ;</li> | |||
| <li>c) applying the changes on the receiving client.</li> | |||
| </ul> | |||
| <p>This works well in general, but it doesn’t take care of conflicts handling, especially when a disconnect can happen.</p> | |||
| <p>For this reason, I got more into “Conflict-free Resolution Data Types” (CRDTs), with the goal of understanding what they are, how they work, what are the different libraries out there, and which one would be a good fit for us, if any.</p> | |||
| <p>As things are changing quickly in this field, note that this article was written in March 2024.</p> | |||
| <hr> | |||
| <div class="toc"> | |||
| <ul> | |||
| <li><a href="#part-1-what-are-crdts">Part 1 - What are CRDTs?</a><ul> | |||
| <li><a href="#why-using-crdts">Why using CRDTs?</a></li> | |||
| <li><a href="#traditional-data-synchronization-methods">Traditional data synchronization methods</a></li> | |||
| <li><a href="#solving-complex-cases">Solving complex cases</a></li> | |||
| <li><a href="#last-write-wins-registers">Last Write Wins Registers</a></li> | |||
| <li><a href="#state-based-vs-operation-based">State-based vs Operation based</a></li> | |||
| <li><a href="#how-the-server-fits-in-the-picture">How the server fits in the picture</a></li> | |||
| <li><a href="#how-offline-is-handled">How offline is handled</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#part-2-javascript-crdts">Part 2: JavaScript CRDTs</a><ul> | |||
| <li><a href="#the-demo-application">The demo application</a></li> | |||
| <li><a href="#yjs">Y.js</a></li> | |||
| <li><a href="#automerge">Automerge</a></li> | |||
| <li><a href="#json-joy"><span class="caps">JSON</span> Joy</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#part-3-comparison-table">Part 3: Comparison table</a><ul> | |||
| <li><a href="#working-with-patches">Working with patches</a></li> | |||
| <li><a href="#conclusion">Conclusion</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#extra-notes">Extra notes</a><ul> | |||
| <li><a href="#yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></a></li> | |||
| <li><a href="#resources">Resources</a></li> | |||
| </ul> | |||
| </li> | |||
| </ul> | |||
| </div> | |||
| <hr> | |||
| <h2 id="part-1-what-are-crdts">Part 1 - What are CRDTs?</h2> | |||
| <p>Conflict-free Resolution Data Types are a family of data types able to merge their states with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.</p> | |||
| <p>CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.</p> | |||
| <p><span class="dquo">“</span>Append-only sets” are probably one of the most common type of <span class="caps">CRDT</span>: you can add the same element again and again, it will only be present once in the set. It’s our old friend <code>Set</code>, as we can find in many programming languages.</p> | |||
| <h3 id="why-using-crdts">Why using CRDTs?</h3> | |||
| <p>For uMap, CRDTs offer a solution to several challenges:</p> | |||
| <ol> | |||
| <li> | |||
| <p><strong>Simultaneous Editing</strong>: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other’s contributions. We need all the replicas to converge to the same state.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Network Latency and Partition</strong>: uMap operates over networks that can experience delays or temporary outages (think editing on the ground). CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Simplified Conflict Resolution</strong>: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Server load</strong>: uMap currently relies on central servers (one per instance). Adopting CRDTs could help lower the work done on the server, increasing resilience and scalability.</p> | |||
| </li> | |||
| </ol> | |||
| <h3 id="traditional-data-synchronization-methods">Traditional data synchronization methods</h3> | |||
| <p>Traditional data synchronization methods typically rely on a central source of truth (the server) to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.</p> | |||
| <p>In contrast, CRDTs leverage mathematical properties (the fact that the data types can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.</p> | |||
| <p>This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.</p> | |||
| <h3 id="solving-complex-cases">Solving complex cases</h3> | |||
| <p>At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures.</p> | |||
| <p>Fortunately for us, our use case is comparatively straightforward, and we probably only need <span class="caps">LWW</span> registers.</p> | |||
| <h3 id="last-write-wins-registers">Last Write Wins Registers</h3> | |||
| <p>As you might have guessed from the name, a <span class="caps">LWW</span> register is a specific type of <span class="caps">CRDT</span> which “just” replaces the value with the last write. The main concern is establishing the sequence of updates, to order them together (who is the last one?).</p> | |||
| <p>In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.</p> | |||
| <p>To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.</p> | |||
| <blockquote> | |||
| <p>A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.</p> | |||
| <p>– <a href="https://en.wikipedia.org/wiki/Vector_clock">Wikipedia</a></p> | |||
| </blockquote> | |||
| <p>Note that we could also use a library such as <a href="https://github.com/pubkey/rxdb">rxdb</a> — to handle the syncing, offline, etc. — because we have a master: we use the server, and we can use it to handle the merge conflicts. | |||
| But by doing so, we would give more responsibility to the server, whereas when using CRDTs it’s possible to do the merge only on the clients (enabling no-master replications).</p> | |||
| <h3 id="state-based-vs-operation-based">State-based vs Operation based</h3> | |||
| <p>While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. It turns out most of the CRDTs implementation I looked at are operation-based, and propose an <span class="caps">API</span> to interact with them as you’re changing the state, so <strong>it doesn’t really matter</strong> in practice.</p> | |||
| <blockquote> | |||
| <p>The two alternatives are theoretically equivalent, as each can emulate the | |||
| other. However, there are practical differences. State-based CRDTs are | |||
| often simpler to design and to implement; their only requirement from the | |||
| communication substrate is some kind of gossip protocol. <strong>Their drawback is that | |||
| the entire state of every <span class="caps">CRDT</span> must be transmitted eventually to every other | |||
| replica, which may be costly</strong>. In contrast, operation-based CRDTs transmit only | |||
| the update operations, which are typically small.</p> | |||
| <p><a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">Wikipedia on CRDTs</a></p> | |||
| </blockquote> | |||
| <h3 id="how-the-server-fits-in-the-picture">How the server fits in the picture</h3> | |||
| <p>While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronization would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers. It makes it easy for the server to send back the needed information to the clients (for new peers, or if the peers didn’t cache the data locally).</p> | |||
| <p>In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be “brainless” and just relay the messages.</p> | |||
| <p>For automerge, it would mean the provider will “just” handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)</p> | |||
| <h3 id="how-offline-is-handled">How offline is handled</h3> | |||
| <p>I was curious about how offline editing might work, and what would happen when going back online. Changes can happen both online and offline, making no difference for the “reconciliation” step. When going back online, a “patch” is computed by the newly reconnected peer, and sent to the other peers.</p> | |||
| <hr> | |||
| <h2 id="part-2-javascript-crdts">Part 2: JavaScript CRDTs</h2> | |||
| <p>Now that we’re familiar with CRDTs and how they can help us, let’s create a map application which syncs marker positions, on different browsers.</p> | |||
| <p>We’ll be comparing three JavaScript libraries: <a href="https://yjs.dev/">Y.js</a>, <a href="https://automerge.org/">Automerge</a> and <a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a>, considering:</p> | |||
| <ol> | |||
| <li><strong>Their external <span class="caps">API</span></strong>: is it easy to use in our case? What are the challenging parts?</li> | |||
| <li><strong>Community and Support</strong>: What is the size and activity of the developer community / ecosystem?</li> | |||
| <li><strong>Size of the <span class="caps">JS</span> library</strong>, because we want to limit the impact on our users browsers.</li> | |||
| <li><strong>Efficiency</strong>: Probe the bandwidth when doing edits. What’s being transmitted over the wire? </li> | |||
| </ol> | |||
| <p>I setup a demo application for each of the libraries. Everything is available <a href="https://gitlab.com/umap-project/leaflet-sync">in a git repository</a> if you want to try it out yourself.</p> | |||
| <h3 id="the-demo-application">The demo application</h3> | |||
| <p>All the demos are made against the same set of features. It</p> | |||
| <ul> | |||
| <li>Creates a marker when the map is clicked</li> | |||
| <li>Moves the markers on hover.</li> | |||
| </ul> | |||
| <p>This should probably be enough for us to try out.</p> | |||
| <p>Here’s the whole code for this, using <a href="https://leafletjs.com/">Leaflet - a JavaScript library for interactive maps</a>. </p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nx">L</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"leaflet"</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="s2">"leaflet/dist/leaflet.css"</span><span class="p">;</span> | |||
| <span class="c1">// Create a map with a default tilelayer.</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="s2">"map"</span><span class="p">).</span><span class="nx">setView</span><span class="p">([</span><span class="mf">48.1173</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="mf">1.6778</span><span class="p">],</span><span class="w"> </span><span class="mf">13</span><span class="p">);</span> | |||
| <span class="nx">L</span><span class="p">.</span><span class="nx">tileLayer</span><span class="p">(</span><span class="s2">"https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">maxZoom</span><span class="o">:</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">attribution</span><span class="o">:</span><span class="w"> </span><span class="s2">"© OpenStreetMap contributors"</span><span class="p">,</span> | |||
| <span class="p">}).</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span> | |||
| <span class="c1">// Features contains a reference to the marker objects, mapped by the uuids</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">features</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span> | |||
| <span class="c1">// An upsert function, creating a marker at the passed latlng.</span> | |||
| <span class="c1">// If an uuid is provided, it changes the coordinates at the given address</span> | |||
| <span class="kd">function</span><span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">uuid</span><span class="p">)</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">crypto</span><span class="p">.</span><span class="nx">randomUUID</span><span class="p">();</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span><span class="w"> </span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">features</span><span class="p">).</span><span class="nx">includes</span><span class="p">(</span><span class="nx">uuid</span><span class="p">))</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">setLatLng</span><span class="p">(</span><span class="nx">latlng</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">marker</span><span class="p">(</span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">draggable</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">autoPan</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">uuid</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">target</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="p">}</span> | |||
| <span class="c1">// Add new features to the map with a click</span> | |||
| <span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>It does the following:</p> | |||
| <ul> | |||
| <li>Creates a map zoomed on Rennes, France</li> | |||
| <li>Maintains a <code>features</code> object, referencing the added markers</li> | |||
| <li>Provides a <code>upsertMarker</code> function, creating a marker on the map at the given latitude and longitude, and updating its latitude and longitude if it already exists.</li> | |||
| <li>It listens to the <code>click</code> event on the map, calling <code>upsertMarker</code> with the appropriate arguments.</li> | |||
| </ul> | |||
| <p>Note that the data is not “reactive” (in the sense of React apps): there is no central state that’s updated and triggers the rendering of the user interface.</p> | |||
| <h3 id="yjs">Y.js</h3> | |||
| <p>Y.js is the first library I’ve looked at, probably because it’s the oldest one, and the more commonly referred to.</p> | |||
| <p>The <span class="caps">API</span> seem to offer what we look for, and provides a way to <a href="https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent">observe changes</a>. Here’s how it works:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="kr">as</span><span class="w"> </span><span class="nx">Y</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"yjs"</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"y-websocket"</span><span class="p">;</span> | |||
| <span class="c1">// Instanciate a document</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">doc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Y</span><span class="p">.</span><span class="nx">Doc</span><span class="p">();</span> | |||
| </code></pre></div> | |||
| <p>When we add a new marker, we update the <span class="caps">CRDT</span> (<code>markers.set</code>).</p> | |||
| <div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">markers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">.</span><span class="nx">getMap</span><span class="p">(</span><span class="s2">"markers"</span><span class="p">);</span> | |||
| <span class="nx">markers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">options</span><span class="p">.</span><span class="nx">uuid</span><span class="p">,</span><span class="w"> </span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>Another connected peer can observe the changes, like this:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">markers</span><span class="p">.</span><span class="nx">observe</span><span class="p">((</span><span class="nx">event</span><span class="p">,</span><span class="w"> </span><span class="nx">transaction</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">transaction</span><span class="p">.</span><span class="nx">local</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">event</span><span class="p">.</span><span class="nx">changes</span><span class="p">.</span><span class="nx">keys</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">change</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">markers</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="k">switch</span><span class="p">(</span><span class="nx">change</span><span class="p">.</span><span class="nx">action</span><span class="p">){</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'add'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'update'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="k">break</span><span class="p">;</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'delete'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Property "</span><span class="si">${</span><span class="nx">key</span><span class="si">}</span><span class="sb">" was deleted. ".`</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>We cycle on the received changes, and then apply them on our map. In the case of an offline peer coming back online after some time, the <code>observe</code> event will be called only once. </p> | |||
| <p>I’m not dealing with the case of marker deletions here, but deleted items are also taken into account. The data isn’t really deleted in this case, but a “tombstone” is used, making it possible to resolve conflicts (for instance, if one people deleted a marker and some other updated it during the same time).</p> | |||
| <p>Y.js comes with multiple “connection providers”, which make it possible to sync with different protocols (there is even <a href="https://github.com/yousefED/matrix-crdt">a way to sync over the matrix protocol</a> 😇).</p> | |||
| <p>More usefully for us, there is <a href="https://github.com/yjs/y-websocket">an implemented protocol for WebSockets</a>. Here is how to use one of these providers:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="c1">// Sync clients with the y-websocket provider</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="p">(</span> | |||
| <span class="w"> </span><span class="s2">"ws://localhost:1234"</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="s2">"leaflet-sync"</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">doc</span> | |||
| <span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>This code setups a WebSocket connection with a server that will maintain a local copy of the <span class="caps">CRDT</span>, as explained above.</p> | |||
| <p>It’s also possible to send “awareness” information (some state you don’t want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers, if they’re connected, etc.</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"mousemove"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">awareness</span><span class="p">.</span><span class="nx">setLocalStateField</span><span class="p">(</span><span class="s2">"user"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">cursor</span><span class="o">:</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span><span class="w"> </span> | |||
| </code></pre></div> | |||
| <p>I made <a href="https://gitlab.com/umap-project/leaflet-sync/-/tree/yjs">a quick proof of concept with Y.js</a> in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.</p> | |||
| <h4 id="python-bindings">Python bindings</h4> | |||
| <p>Y.js has been ported to rust with <a href="https://github.com/y-crdt/y-crdt">the Y.rs project</a>, making it possible to have binding in other languages, like ruby and python (see <a href="https://github.com/y-crdt/ypy">Y.py</a>). The python bindings are currently looking for a maintainer.</p> | |||
| <h4 id="library-size">Library size</h4> | |||
| <ul> | |||
| <li>Y.js is 4,16 Ko</li> | |||
| <li>Y-Websocket is 21,14 Ko</li> | |||
| </ul> | |||
| <h4 id="the-data-being-transmitted">The data being transmitted</h4> | |||
| <p>In a scenario where all clients connect to a central server, which handles the <span class="caps">CRDT</span> locally and then transmits back to other parties, adding 20 points on one client, then 20 points in another generates ~5 Ko of data (so approximately 16 bytes per edit).</p> | |||
| <p>Pros: </p> | |||
| <ul> | |||
| <li>The <span class="caps">API</span> was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate.</li> | |||
| <li>It seems to be widely used, and the community seems active.</li> | |||
| <li>It is <a href="https://docs.yjs.dev/">well documented</a></li> | |||
| <li>There is awareness support</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>It doesn’t seem to work well <a href="https://github.com/yjs/yjs/issues/325">without a <span class="caps">JS</span> bundler</a> which could be a problem for us.</li> | |||
| <li>The Websocket connection provider doesn’t do what I was expecting it to, as it requires the server to run a <span class="caps">CRDT</span> locally.</li> | |||
| </ul> | |||
| <hr> | |||
| <h3 id="automerge">Automerge</h3> | |||
| <p><a href="https://automerge.org/">Automerge</a> is another <span class="caps">CRDT</span> library started by the folks at <a href="https://www.inkandswitch.com/">Ink <span class="amp">&</span> Switch</a> with Martin Kleppmann. Automerge is actually the low level interface. There is a higher-level interface named <a href="https://automerge.org/docs/repositories/">Automerge-repo</a>. Here is how to use it:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">Repo</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@automerge/automerge-repo"</span><span class="p">;</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">repo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Repo</span><span class="p">();</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">repo</span><span class="p">.</span><span class="nx">create</span><span class="p">()</span> | |||
| <span class="c1">// or repo.find(name)</span> | |||
| </code></pre></div> | |||
| <p>To change the document, call <code>handle.change</code> and pass it a function that will make changes to the document.</p> | |||
| <p>Here, when a new marker is added:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">change</span><span class="p">((</span><span class="nx">doc</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">doc</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">cleanLatLng</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>I had to use a <code>cleanLatLng</code> function in order to not pass the whole object, otherwise it wouldn’t serialize. It’s really just a simple helper taking the properties of interest for us (and letting away all the rest).</p> | |||
| <p>Another peer can observe the changes, like this:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"change"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">doc</span><span class="p">,</span><span class="w"> </span><span class="nx">patches</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">patches</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(({</span><span class="w"> </span><span class="nx">action</span><span class="p">,</span><span class="w"> </span><span class="nx">path</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mf">2</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"insert"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">[</span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">]];</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">],</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>The “patch” interface is a bit less verbose than the one from Y.js. It wasn’t well documented, so I had to play around with the messages to understand exactly what the possible actions were. In the end, it gets me what I’m looking for, the changes that occurred to my data.</p> | |||
| <p>Here I’m using <code>path.length == 2 && action === 'insert'</code> to detect that a change occurred to the marker. Since we don’t make a difference between the creation of a marker and its update (when being moved), it works well for us.</p> | |||
| <h4 id="python-bindings_1">Python bindings</h4> | |||
| <p>There are <a href="https://github.com/automerge/automerge-py">python bindings</a> for automerge “core”. (It doesn’t — yet — provide ways to interact with “repos”).</p> | |||
| <h4 id="library-size_1">Library size</h4> | |||
| <ul> | |||
| <li>Size of the automerge + automerge repo: 1,64 mb</li> | |||
| <li>Size of the WebSocket provider: 0,10 mb </li> | |||
| </ul> | |||
| <p>This is quite a large bundle size, and the team behind automerge is aware of it and working on a solution.</p> | |||
| <h4 id="the-data-being-transmitted_1">The data being transmitted</h4> | |||
| <p>In the same scenario, adding 20 points on one client, then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.</p> | |||
| <p>Pros:</p> | |||
| <ul> | |||
| <li>There is an <span class="caps">API</span> to <a href="https://automerge.org/docs/documents/conflicts/">get informed when a conflict occurred</a></li> | |||
| <li>Python bindings are currently being worked on, soon to reach a stable version</li> | |||
| <li>The team was overall very responsive and trying to help.</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>The JavaScript is currently generated via Web Assembly, which could make it harder to debug.</li> | |||
| <li>The large bundle size of the generated files.</li> | |||
| </ul> | |||
| <hr> | |||
| <h3 id="json-joy"><span class="caps">JSON</span> Joy</h3> | |||
| <p><a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a> is the latest to the party.</p> | |||
| <p>It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would leave us with the hands free in order to implement the protocol that would work for us.</p> | |||
| <p>Here is how to use it. On the different peers you start with different forks of the same document:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="nx">Model</span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s1">'json-joy/es2020/json-crdt'</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">s</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch"</span><span class="p">;</span> | |||
| <span class="c1">// Initiate a model with a custom ID</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">rootModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Model</span><span class="p">.</span><span class="nx">withLogicalClock</span><span class="p">(</span><span class="mf">11111111</span><span class="p">);</span> | |||
| <span class="c1">// populate it with default data</span> | |||
| <span class="nx">rootModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">root</span><span class="p">({</span> | |||
| <span class="w"> </span><span class="nx">markers</span><span class="o">:</span><span class="w"> </span><span class="p">{},</span> | |||
| <span class="p">});</span> | |||
| <span class="c1">// Fork it on each client</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">userModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">rootModel</span><span class="p">.</span><span class="nx">fork</span><span class="p">();</span> | |||
| </code></pre></div> | |||
| <p>When adding a new marker, we can define a new constant, by using <code>s.con</code>…</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">obj</span><span class="p">([</span><span class="s2">"markers"</span><span class="p">]).</span><span class="nx">set</span><span class="p">({</span> | |||
| <span class="w"> </span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="o">:</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">con</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">),</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>… and then create a patch and send it to the other peers:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">encode</span><span class="p">,</span><span class="w"> </span><span class="nx">decode</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch/codec/verbose"</span><span class="p">;</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">flush</span><span class="p">();</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">payload</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">encode</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>On the other peers, when we receive a patch message, decode it and apply it:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">decode</span><span class="p">(</span><span class="nx">payload</span><span class="p">);</span> | |||
| <span class="nx">model</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>We can observe the changes this way:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">onPatch</span><span class="p">.</span><span class="nx">listen</span><span class="p">((</span><span class="nx">patch</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">patch</span><span class="p">.</span><span class="nx">ops</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">op</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">op</span><span class="p">.</span><span class="nx">name</span><span class="p">()</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"ins_obj"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">op</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="mf">0</span><span class="p">][</span><span class="mf">0</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">view</span><span class="p">().</span><span class="nx">markers</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>Similarly to what we did with automerge, we’re having a look into the patch, and filter on the operations of interest for us (<code>ins_obj</code>). The names of the operations aren’t clearly specified by the spec.</p> | |||
| <p>Metrics:</p> | |||
| <ul> | |||
| <li>Size: 143 ko</li> | |||
| <li>Data transmitted for 2 peers and 40 edits: (35 bytes per edit)</li> | |||
| </ul> | |||
| <p>Pros:</p> | |||
| <ul> | |||
| <li>Small atomic libraries, making it easy to use only the parts we need.</li> | |||
| <li>The interface <a href="https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types">proposes to store different type of data</a> (constants, values, arrays, etc.)</li> | |||
| <li>Distributed as different type of <span class="caps">JS</span> bundles (modules, wasm, etc.)</li> | |||
| <li>Low level, so you know what you’re doing</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>It doesn’t provide a high level interface for sync</li> | |||
| <li>It’s currently a one-person project, without clear community channels to gather with other interested folks.</li> | |||
| <li>Quite recent, so probably rough spots are to be found</li> | |||
| </ul> | |||
| <hr> | |||
| <h2 id="part-3-comparison-table">Part 3: Comparison table</h2> | |||
| <p>I found Y.js and Automerge quite similar for my use case, while <span class="caps">JSON</span> Joy was taking a different (less “all-included”) approach. Here is a summary table to help read the differences I found.</p> | |||
| <table> | |||
| <thead> | |||
| <tr> | |||
| <th></th> | |||
| <th>Y.js</th> | |||
| <th>Automerge</th> | |||
| <th><span class="caps">JSON</span> Joy</th> | |||
| </tr> | |||
| </thead> | |||
| <tbody> | |||
| <tr> | |||
| <td>Python bindings</td> | |||
| <td><a href="https://github.com/y-crdt/ypy">Yes</a></td> | |||
| <td><a href="https://github.com/automerge/automerge-py">Yes</a></td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Syncing</td> | |||
| <td>Native <span class="caps">JS</span> structures</td> | |||
| <td>Transactional functions</td> | |||
| <td>Specific types (bonus points for handling constants)</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Coded in</td> | |||
| <td>JavaScript / Rust</td> | |||
| <td>TypeScript / Rust</td> | |||
| <td>Typescript</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Awareness protocol</td> | |||
| <td>Yes, with presence</td> | |||
| <td>Yes, without presence</td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Conflict-detection <span class="caps">API</span></td> | |||
| <td>No</td> | |||
| <td>Yes</td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Library size</td> | |||
| <td><strong>24.3Ko</strong> §<br></td> | |||
| <td><strong>1,74 mb</strong> §</td> | |||
| <td><strong>143 ko</strong></td> | |||
| </tr> | |||
| </tbody> | |||
| </table> | |||
| <p>§ size of the connectors included</p> | |||
| <h3 id="working-with-patches">Working with patches</h3> | |||
| <p>In order to observe the changes, we need to inspect the given patches and work on what we find. I found the different libraries expose different sets of APIs. All of these APIs were quite a bit hard to find, and it’s not clear if they are public or private.</p> | |||
| <p>One thing to keep in mind is that these “patch” events happen only once per patch received. You can see it as a “diff” of the state between the current and the incoming states.</p> | |||
| <ul> | |||
| <li>Y.js exposes a utility which is able to tell you what the action on the key is (“delete”, “update” and “add”)</li> | |||
| <li>Automerge and <span class="caps">JSON</span> Joy, on the other hand, don’t provide such utility functions, meaning you would need to find that out yourself.</li> | |||
| </ul> | |||
| <h3 id="conclusion">Conclusion</h3> | |||
| <p>The goal here is not to tell which one of these libraries is the best one. They’re all great and have their strenghs. None of them implement the high-level <span class="caps">API</span> I was expecting, where the clients talk with each other and the server just relays messages, but maybe it’s because it’s better in general to have the server have the representation of the data, saving a roundtrip for the clients.</p> | |||
| <p>I wasn’t expecting to have a look at patches to understand what changed at the low level. The way it’s currently implemented is very suitable for “reactive” applications, but require more involvement to sync between the CRDTs state and the application state.</p> | |||
| <p>In the end, adding CRDTs is made very simple, probably due to the fact all we really need is a sort of distributed key/value store.</p> | |||
| <p>I’m not sure yet which library we will end up using for uMap (if any), but my understanding is clearer than it was when I started. I guess that’s what progress looks like 😎</p> | |||
| <h3 id="yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></h3> | |||
| <p>While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:</p> | |||
| <blockquote> | |||
| <ul> | |||
| <li><strong><span class="caps">RGA</span></strong> [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that’s updated anytime we detect that remote insert has an id | |||
| with sequence number higher that local counter. Therefore, every time we produce | |||
| a new insert operation, we give it a highest counter value known at the time.</li> | |||
| <li><strong><span class="caps">YATA</span></strong> [used by Yjs] also uses a single integer value, however unlike in case of <span class="caps">RGA</span> we | |||
| don’t use a single counter shared with other replicas, but rather let each | |||
| peer keep its own, which is incremented monotonically only by that peer. Since | |||
| increments are monotonic, we can also use them to detect missing operations eg. | |||
| updates marked as A:1 and A:3 imply, that there must be another (potentially | |||
| missing) update A:2.</li> | |||
| </ul> | |||
| </blockquote> | |||
| <h3 id="resources">Resources</h3> | |||
| <ul> | |||
| <li><a href="https://www.youtube.com/watch?v=x7drE24geUw">CRDTs: The Hard Parts</a>, a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren’t solved yet.</li> | |||
| <li><a href="https://jakelazaroff.com/words/an-interactive-intro-to-crdts/">An Interactive Intro to CRDTs</a> gets you trough different steps to understand what are CRDTs, and how to implement a <span class="caps">LWW</span> Register.</li> | |||
| <li><a href="https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/">Bartosz Sypytkowski</a> introduction on CRDTs, with practical examples is very intuitive.</li> | |||
| <li><a href="https://jzhao.xyz/thoughts/CRDT-Implementations"><span class="caps">CRDT</span> Implementations</a> are notes by Jacky which were useful to me when understanding CRDTs.</li> | |||
| </ul> | |||
| </article> | |||
| <hr> | |||
| <footer> | |||
| <p> | |||
| <a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use> | |||
| </svg> Accueil</a> • | |||
| <a href="/david/log/" title="Accès au flux RSS"><svg class="icon icon-rss2"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-rss2"></use> | |||
| </svg> Suivre</a> • | |||
| <a href="http://larlet.com" title="Go to my English profile" data-instant><svg class="icon icon-user-tie"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-user-tie"></use> | |||
| </svg> Pro</a> • | |||
| <a href="mailto:david%40larlet.fr" title="Envoyer un courriel"><svg class="icon icon-mail"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-mail"></use> | |||
| </svg> Email</a> • | |||
| <abbr class="nowrap" title="Hébergeur : Alwaysdata, 62 rue Tiquetonne 75002 Paris, +33184162340"><svg class="icon icon-hammer2"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-hammer2"></use> | |||
| </svg> Légal</abbr> | |||
| </p> | |||
| <template id="theme-selector"> | |||
| <form> | |||
| <fieldset> | |||
| <legend><svg class="icon icon-brightness-contrast"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-brightness-contrast"></use> | |||
| </svg> Thème</legend> | |||
| <label> | |||
| <input type="radio" value="auto" name="chosen-color-scheme" checked> Auto | |||
| </label> | |||
| <label> | |||
| <input type="radio" value="dark" name="chosen-color-scheme"> Foncé | |||
| </label> | |||
| <label> | |||
| <input type="radio" value="light" name="chosen-color-scheme"> Clair | |||
| </label> | |||
| </fieldset> | |||
| </form> | |||
| </template> | |||
| </footer> | |||
| <script src="/static/david/js/instantpage-5.1.0.min.js" type="module"></script> | |||
| <script> | |||
| function loadThemeForm(templateName) { | |||
| const themeSelectorTemplate = document.querySelector(templateName) | |||
| const form = themeSelectorTemplate.content.firstElementChild | |||
| themeSelectorTemplate.replaceWith(form) | |||
| form.addEventListener('change', (e) => { | |||
| const chosenColorScheme = e.target.value | |||
| localStorage.setItem('theme', chosenColorScheme) | |||
| toggleTheme(chosenColorScheme) | |||
| }) | |||
| const selectedTheme = localStorage.getItem('theme') | |||
| if (selectedTheme && selectedTheme !== 'undefined') { | |||
| form.querySelector(`[value="${selectedTheme}"]`).checked = true | |||
| } | |||
| } | |||
| const prefersColorSchemeDark = '(prefers-color-scheme: dark)' | |||
| window.addEventListener('load', () => { | |||
| let hasDarkRules = false | |||
| for (const styleSheet of Array.from(document.styleSheets)) { | |||
| let mediaRules = [] | |||
| for (const cssRule of styleSheet.cssRules) { | |||
| if (cssRule.type !== CSSRule.MEDIA_RULE) { | |||
| continue | |||
| } | |||
| // WARNING: Safari does not have/supports `conditionText`. | |||
| if (cssRule.conditionText) { | |||
| if (cssRule.conditionText !== prefersColorSchemeDark) { | |||
| continue | |||
| } | |||
| } else { | |||
| if (cssRule.cssText.startsWith(prefersColorSchemeDark)) { | |||
| continue | |||
| } | |||
| } | |||
| mediaRules = mediaRules.concat(Array.from(cssRule.cssRules)) | |||
| } | |||
| // WARNING: do not try to insert a Rule to a styleSheet you are | |||
| // currently iterating on, otherwise the browser will be stuck | |||
| // in a infinite loop… | |||
| for (const mediaRule of mediaRules) { | |||
| styleSheet.insertRule(mediaRule.cssText) | |||
| hasDarkRules = true | |||
| } | |||
| } | |||
| if (hasDarkRules) { | |||
| loadThemeForm('#theme-selector') | |||
| } | |||
| }) | |||
| </script> | |||
| </body> | |||
| </html> | |||
+ 465
- 0
cache/2024/2c0b2588dfcd3a194da4133c7505cd3e/index.md
View File
| @@ -0,0 +1,465 @@ | |||
| title: A comparison of JavaScript CRDTs | |||
| url: https://blog.notmyidea.org/a-comparison-of-javascript-crdts.html | |||
| hash_url: 2c0b2588dfcd3a194da4133c7505cd3e | |||
| archive_date: 2024-03-25 | |||
| og_image: https://blog.notmyidea.org/images/umap/crdt-converge.png | |||
| description: | |||
| favicon: https://blog.notmyidea.org/favicon-32x32.png | |||
| language: fr_FR | |||
| <p>Collaboration is one of the most requested features on <a href="https://umap-project.org">uMap</a>. | |||
| I’ve talked <a href="https://blog.notmyidea.org/tag/umap.html">in previous articles</a> how | |||
| we could add real-time features “the simple way”, by:</p> | |||
| <ul> | |||
| <li>a) catching when changes are done on the interface ;</li> | |||
| <li>b) sending messages to the other parties and ;</li> | |||
| <li>c) applying the changes on the receiving client.</li> | |||
| </ul> | |||
| <p>This works well in general, but it doesn’t take care of conflicts handling, especially when a disconnect can happen.</p> | |||
| <p>For this reason, I got more into “Conflict-free Resolution Data Types” (CRDTs), with the goal of understanding what they are, how they work, what are the different libraries out there, and which one would be a good fit for us, if any.</p> | |||
| <p>As things are changing quickly in this field, note that this article was written in March 2024.</p> | |||
| <hr> | |||
| <div class="toc"> | |||
| <ul> | |||
| <li><a href="#part-1-what-are-crdts">Part 1 - What are CRDTs?</a><ul> | |||
| <li><a href="#why-using-crdts">Why using CRDTs?</a></li> | |||
| <li><a href="#traditional-data-synchronization-methods">Traditional data synchronization methods</a></li> | |||
| <li><a href="#solving-complex-cases">Solving complex cases</a></li> | |||
| <li><a href="#last-write-wins-registers">Last Write Wins Registers</a></li> | |||
| <li><a href="#state-based-vs-operation-based">State-based vs Operation based</a></li> | |||
| <li><a href="#how-the-server-fits-in-the-picture">How the server fits in the picture</a></li> | |||
| <li><a href="#how-offline-is-handled">How offline is handled</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#part-2-javascript-crdts">Part 2: JavaScript CRDTs</a><ul> | |||
| <li><a href="#the-demo-application">The demo application</a></li> | |||
| <li><a href="#yjs">Y.js</a></li> | |||
| <li><a href="#automerge">Automerge</a></li> | |||
| <li><a href="#json-joy"><span class="caps">JSON</span> Joy</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#part-3-comparison-table">Part 3: Comparison table</a><ul> | |||
| <li><a href="#working-with-patches">Working with patches</a></li> | |||
| <li><a href="#conclusion">Conclusion</a></li> | |||
| </ul> | |||
| </li> | |||
| <li><a href="#extra-notes">Extra notes</a><ul> | |||
| <li><a href="#yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></a></li> | |||
| <li><a href="#resources">Resources</a></li> | |||
| </ul> | |||
| </li> | |||
| </ul> | |||
| </div> | |||
| <hr> | |||
| <h2 id="part-1-what-are-crdts">Part 1 - What are CRDTs?</h2> | |||
| <p>Conflict-free Resolution Data Types are a family of data types able to merge their states with other states without generating conflicts. They handle consistency in distributed systems, making them particularly well-suited for collaborative real-time applications.</p> | |||
| <p>CRDTs ensure that multiple participants can make changes without strict coordination, and all replicas converge to the same state upon synchronization, without conflicts.</p> | |||
| <p><span class="dquo">“</span>Append-only sets” are probably one of the most common type of <span class="caps">CRDT</span>: you can add the same element again and again, it will only be present once in the set. It’s our old friend <code>Set</code>, as we can find in many programming languages.</p> | |||
| <h3 id="why-using-crdts">Why using CRDTs?</h3> | |||
| <p>For uMap, CRDTs offer a solution to several challenges:</p> | |||
| <ol> | |||
| <li> | |||
| <p><strong>Simultaneous Editing</strong>: When multiple users interact with the same map, their changes must not only be reflected in real-time but also merged seamlessly without overwriting each other’s contributions. We need all the replicas to converge to the same state.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Network Latency and Partition</strong>: uMap operates over networks that can experience delays or temporary outages (think editing on the ground). CRDTs can handle these conditions gracefully, enabling offline editing and eventual consistency.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Simplified Conflict Resolution</strong>: Traditional methods often require complex algorithms to resolve conflicts, while CRDTs inherently minimize the occurrence of conflicts altogether.</p> | |||
| </li> | |||
| <li> | |||
| <p><strong>Server load</strong>: uMap currently relies on central servers (one per instance). Adopting CRDTs could help lower the work done on the server, increasing resilience and scalability.</p> | |||
| </li> | |||
| </ol> | |||
| <h3 id="traditional-data-synchronization-methods">Traditional data synchronization methods</h3> | |||
| <p>Traditional data synchronization methods typically rely on a central source of truth (the server) to manage and resolve conflicts. When changes are made by different users, these traditional systems require a round-trip to the server for conflict resolution and thus can be slow or inadequate for real-time collaboration.</p> | |||
| <p>In contrast, CRDTs leverage mathematical properties (the fact that the data types can converge) to ensure that every replica independently reaches the same state, without the need for a central authority, thus minimizing the amount of coordination and communication needed between nodes.</p> | |||
| <p>This ability to maintain consistency sets CRDTs apart from conventional synchronization approaches and makes them particularly valuable for the development of collaborative tools like uMap, where real-time updates and reliability are important.</p> | |||
| <h3 id="solving-complex-cases">Solving complex cases</h3> | |||
| <p>At first, I found CRDTs somewhat confusing, owing to their role in addressing complex challenges. CRDTs come in various forms, with much of their intricacy tied to resolving content conflicts within textual data or elaborate hierarchical structures.</p> | |||
| <p>Fortunately for us, our use case is comparatively straightforward, and we probably only need <span class="caps">LWW</span> registers.</p> | |||
| <h3 id="last-write-wins-registers">Last Write Wins Registers</h3> | |||
| <p>As you might have guessed from the name, a <span class="caps">LWW</span> register is a specific type of <span class="caps">CRDT</span> which “just” replaces the value with the last write. The main concern is establishing the sequence of updates, to order them together (who is the last one?).</p> | |||
| <p>In a single-client scenario or with a central time reference, sequencing is straightforward. However, in a distributed environment, time discrepancies across peers can complicate things, as clocks may drift and lose synchronization.</p> | |||
| <p>To address this, CRDTs use vector clocks — a specialized data structure that helps to solve the relative timing of events across distributed systems and pinpoint any inconsistencies.</p> | |||
| <blockquote> | |||
| <p>A vector clock is a data structure used for determining the partial ordering of events in a distributed system and detecting causality violations.</p> | |||
| <p>– <a href="https://en.wikipedia.org/wiki/Vector_clock">Wikipedia</a></p> | |||
| </blockquote> | |||
| <p>Note that we could also use a library such as <a href="https://github.com/pubkey/rxdb">rxdb</a> — to handle the syncing, offline, etc. — because we have a master: we use the server, and we can use it to handle the merge conflicts. | |||
| But by doing so, we would give more responsibility to the server, whereas when using CRDTs it’s possible to do the merge only on the clients (enabling no-master replications).</p> | |||
| <h3 id="state-based-vs-operation-based">State-based vs Operation based</h3> | |||
| <p>While reading the literature, I found that there are two kinds of CRDTs: state-based and operation-based. It turns out most of the CRDTs implementation I looked at are operation-based, and propose an <span class="caps">API</span> to interact with them as you’re changing the state, so <strong>it doesn’t really matter</strong> in practice.</p> | |||
| <blockquote> | |||
| <p>The two alternatives are theoretically equivalent, as each can emulate the | |||
| other. However, there are practical differences. State-based CRDTs are | |||
| often simpler to design and to implement; their only requirement from the | |||
| communication substrate is some kind of gossip protocol. <strong>Their drawback is that | |||
| the entire state of every <span class="caps">CRDT</span> must be transmitted eventually to every other | |||
| replica, which may be costly</strong>. In contrast, operation-based CRDTs transmit only | |||
| the update operations, which are typically small.</p> | |||
| <p><a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">Wikipedia on CRDTs</a></p> | |||
| </blockquote> | |||
| <h3 id="how-the-server-fits-in-the-picture">How the server fits in the picture</h3> | |||
| <p>While discussing with the automerge team, I understood that I was expecting the server to pass along the messages to the other parties, and that would be the way the synchronization would be done. It turns out I was mistaken: in this approach, the clients send updates to the server, which merges everything together and only then sends the updates to the other peers. It makes it easy for the server to send back the needed information to the clients (for new peers, or if the peers didn’t cache the data locally).</p> | |||
| <p>In order to have peers working with each other, I would need to change the way the provider works, so we can have the server be “brainless” and just relay the messages.</p> | |||
| <p>For automerge, it would mean the provider will “just” handle the websocket connection (disconnect and reconnect) and all the peers would be able to talk with each other. The other solution for us would be to have the merge algorithm working on the server side, which comes with upsides (no need to find when the document should be saved by the client to the server) and downsides (it takes some cpu and memory to run the CRDTs on the server)</p> | |||
| <h3 id="how-offline-is-handled">How offline is handled</h3> | |||
| <p>I was curious about how offline editing might work, and what would happen when going back online. Changes can happen both online and offline, making no difference for the “reconciliation” step. When going back online, a “patch” is computed by the newly reconnected peer, and sent to the other peers.</p> | |||
| <hr> | |||
| <h2 id="part-2-javascript-crdts">Part 2: JavaScript CRDTs</h2> | |||
| <p>Now that we’re familiar with CRDTs and how they can help us, let’s create a map application which syncs marker positions, on different browsers.</p> | |||
| <p>We’ll be comparing three JavaScript libraries: <a href="https://yjs.dev/">Y.js</a>, <a href="https://automerge.org/">Automerge</a> and <a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a>, considering:</p> | |||
| <ol> | |||
| <li><strong>Their external <span class="caps">API</span></strong>: is it easy to use in our case? What are the challenging parts?</li> | |||
| <li><strong>Community and Support</strong>: What is the size and activity of the developer community / ecosystem?</li> | |||
| <li><strong>Size of the <span class="caps">JS</span> library</strong>, because we want to limit the impact on our users browsers.</li> | |||
| <li><strong>Efficiency</strong>: Probe the bandwidth when doing edits. What’s being transmitted over the wire? </li> | |||
| </ol> | |||
| <p>I setup a demo application for each of the libraries. Everything is available <a href="https://gitlab.com/umap-project/leaflet-sync">in a git repository</a> if you want to try it out yourself.</p> | |||
| <h3 id="the-demo-application">The demo application</h3> | |||
| <p>All the demos are made against the same set of features. It</p> | |||
| <ul> | |||
| <li>Creates a marker when the map is clicked</li> | |||
| <li>Moves the markers on hover.</li> | |||
| </ul> | |||
| <p>This should probably be enough for us to try out.</p> | |||
| <p>Here’s the whole code for this, using <a href="https://leafletjs.com/">Leaflet - a JavaScript library for interactive maps</a>. </p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="nx">L</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"leaflet"</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="s2">"leaflet/dist/leaflet.css"</span><span class="p">;</span> | |||
| <span class="c1">// Create a map with a default tilelayer.</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="s2">"map"</span><span class="p">).</span><span class="nx">setView</span><span class="p">([</span><span class="mf">48.1173</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="mf">1.6778</span><span class="p">],</span><span class="w"> </span><span class="mf">13</span><span class="p">);</span> | |||
| <span class="nx">L</span><span class="p">.</span><span class="nx">tileLayer</span><span class="p">(</span><span class="s2">"https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">maxZoom</span><span class="o">:</span><span class="w"> </span><span class="mf">19</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">attribution</span><span class="o">:</span><span class="w"> </span><span class="s2">"© OpenStreetMap contributors"</span><span class="p">,</span> | |||
| <span class="p">}).</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span> | |||
| <span class="c1">// Features contains a reference to the marker objects, mapped by the uuids</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">features</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{};</span> | |||
| <span class="c1">// An upsert function, creating a marker at the passed latlng.</span> | |||
| <span class="c1">// If an uuid is provided, it changes the coordinates at the given address</span> | |||
| <span class="kd">function</span><span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">uuid</span><span class="p">)</span><span class="w"> </span><span class="nx">uuid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">crypto</span><span class="p">.</span><span class="nx">randomUUID</span><span class="p">();</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span><span class="w"> </span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">features</span><span class="p">).</span><span class="nx">includes</span><span class="p">(</span><span class="nx">uuid</span><span class="p">))</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">setLatLng</span><span class="p">(</span><span class="nx">latlng</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">L</span><span class="p">.</span><span class="nx">marker</span><span class="p">(</span><span class="nx">latlng</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">draggable</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">autoPan</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">uuid</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="nx">features</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">marker</span><span class="p">;</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="nx">marker</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">target</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">"dragend"</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="p">}</span> | |||
| <span class="c1">// Add new features to the map with a click</span> | |||
| <span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"click"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>It does the following:</p> | |||
| <ul> | |||
| <li>Creates a map zoomed on Rennes, France</li> | |||
| <li>Maintains a <code>features</code> object, referencing the added markers</li> | |||
| <li>Provides a <code>upsertMarker</code> function, creating a marker on the map at the given latitude and longitude, and updating its latitude and longitude if it already exists.</li> | |||
| <li>It listens to the <code>click</code> event on the map, calling <code>upsertMarker</code> with the appropriate arguments.</li> | |||
| </ul> | |||
| <p>Note that the data is not “reactive” (in the sense of React apps): there is no central state that’s updated and triggers the rendering of the user interface.</p> | |||
| <h3 id="yjs">Y.js</h3> | |||
| <p>Y.js is the first library I’ve looked at, probably because it’s the oldest one, and the more commonly referred to.</p> | |||
| <p>The <span class="caps">API</span> seem to offer what we look for, and provides a way to <a href="https://docs.yjs.dev/api/shared-types/y.map#observing-changes-y.mapevent">observe changes</a>. Here’s how it works:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="kr">as</span><span class="w"> </span><span class="nx">Y</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"yjs"</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"y-websocket"</span><span class="p">;</span> | |||
| <span class="c1">// Instanciate a document</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">doc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Y</span><span class="p">.</span><span class="nx">Doc</span><span class="p">();</span> | |||
| </code></pre></div> | |||
| <p>When we add a new marker, we update the <span class="caps">CRDT</span> (<code>markers.set</code>).</p> | |||
| <div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">markers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">.</span><span class="nx">getMap</span><span class="p">(</span><span class="s2">"markers"</span><span class="p">);</span> | |||
| <span class="nx">markers</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">options</span><span class="p">.</span><span class="nx">uuid</span><span class="p">,</span><span class="w"> </span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>Another connected peer can observe the changes, like this:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">markers</span><span class="p">.</span><span class="nx">observe</span><span class="p">((</span><span class="nx">event</span><span class="p">,</span><span class="w"> </span><span class="nx">transaction</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="o">!</span><span class="nx">transaction</span><span class="p">.</span><span class="nx">local</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">event</span><span class="p">.</span><span class="nx">changes</span><span class="p">.</span><span class="nx">keys</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">change</span><span class="p">,</span><span class="w"> </span><span class="nx">key</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">markers</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="k">switch</span><span class="p">(</span><span class="nx">change</span><span class="p">.</span><span class="nx">action</span><span class="p">){</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'add'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'update'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="k">break</span><span class="p">;</span> | |||
| <span class="w"> </span><span class="k">case</span><span class="w"> </span><span class="s1">'delete'</span><span class="o">:</span> | |||
| <span class="w"> </span><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Property "</span><span class="si">${</span><span class="nx">key</span><span class="si">}</span><span class="sb">" was deleted. ".`</span><span class="p">);</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>We cycle on the received changes, and then apply them on our map. In the case of an offline peer coming back online after some time, the <code>observe</code> event will be called only once. </p> | |||
| <p>I’m not dealing with the case of marker deletions here, but deleted items are also taken into account. The data isn’t really deleted in this case, but a “tombstone” is used, making it possible to resolve conflicts (for instance, if one people deleted a marker and some other updated it during the same time).</p> | |||
| <p>Y.js comes with multiple “connection providers”, which make it possible to sync with different protocols (there is even <a href="https://github.com/yousefED/matrix-crdt">a way to sync over the matrix protocol</a> 😇).</p> | |||
| <p>More usefully for us, there is <a href="https://github.com/yjs/y-websocket">an implemented protocol for WebSockets</a>. Here is how to use one of these providers:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="c1">// Sync clients with the y-websocket provider</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">provider</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">WebsocketProvider</span><span class="p">(</span> | |||
| <span class="w"> </span><span class="s2">"ws://localhost:1234"</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="s2">"leaflet-sync"</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="nx">doc</span> | |||
| <span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>This code setups a WebSocket connection with a server that will maintain a local copy of the <span class="caps">CRDT</span>, as explained above.</p> | |||
| <p>It’s also possible to send “awareness” information (some state you don’t want to persist, like the position of the cursor). It contains some useful meta information, such as the number of connected peers, if they’re connected, etc.</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">map</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"mousemove"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">awareness</span><span class="p">.</span><span class="nx">setLocalStateField</span><span class="p">(</span><span class="s2">"user"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">cursor</span><span class="o">:</span><span class="w"> </span><span class="nx">latlng</span><span class="p">,</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span><span class="w"> </span> | |||
| </code></pre></div> | |||
| <p>I made <a href="https://gitlab.com/umap-project/leaflet-sync/-/tree/yjs">a quick proof of concept with Y.js</a> in a few hours flawlessly. It handles offline and reconnects, and exposes awareness information.</p> | |||
| <h4 id="python-bindings">Python bindings</h4> | |||
| <p>Y.js has been ported to rust with <a href="https://github.com/y-crdt/y-crdt">the Y.rs project</a>, making it possible to have binding in other languages, like ruby and python (see <a href="https://github.com/y-crdt/ypy">Y.py</a>). The python bindings are currently looking for a maintainer.</p> | |||
| <h4 id="library-size">Library size</h4> | |||
| <ul> | |||
| <li>Y.js is 4,16 Ko</li> | |||
| <li>Y-Websocket is 21,14 Ko</li> | |||
| </ul> | |||
| <h4 id="the-data-being-transmitted">The data being transmitted</h4> | |||
| <p>In a scenario where all clients connect to a central server, which handles the <span class="caps">CRDT</span> locally and then transmits back to other parties, adding 20 points on one client, then 20 points in another generates ~5 Ko of data (so approximately 16 bytes per edit).</p> | |||
| <p>Pros: </p> | |||
| <ul> | |||
| <li>The <span class="caps">API</span> was feeling natural to me: it handles plain old JavaScript objects, making it easy to integrate.</li> | |||
| <li>It seems to be widely used, and the community seems active.</li> | |||
| <li>It is <a href="https://docs.yjs.dev/">well documented</a></li> | |||
| <li>There is awareness support</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>It doesn’t seem to work well <a href="https://github.com/yjs/yjs/issues/325">without a <span class="caps">JS</span> bundler</a> which could be a problem for us.</li> | |||
| <li>The Websocket connection provider doesn’t do what I was expecting it to, as it requires the server to run a <span class="caps">CRDT</span> locally.</li> | |||
| </ul> | |||
| <hr> | |||
| <h3 id="automerge">Automerge</h3> | |||
| <p><a href="https://automerge.org/">Automerge</a> is another <span class="caps">CRDT</span> library started by the folks at <a href="https://www.inkandswitch.com/">Ink <span class="amp">&</span> Switch</a> with Martin Kleppmann. Automerge is actually the low level interface. There is a higher-level interface named <a href="https://automerge.org/docs/repositories/">Automerge-repo</a>. Here is how to use it:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">Repo</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"@automerge/automerge-repo"</span><span class="p">;</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">repo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nx">Repo</span><span class="p">();</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">handle</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">repo</span><span class="p">.</span><span class="nx">create</span><span class="p">()</span> | |||
| <span class="c1">// or repo.find(name)</span> | |||
| </code></pre></div> | |||
| <p>To change the document, call <code>handle.change</code> and pass it a function that will make changes to the document.</p> | |||
| <p>Here, when a new marker is added:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">change</span><span class="p">((</span><span class="nx">doc</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">doc</span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">cleanLatLng</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">);</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>I had to use a <code>cleanLatLng</code> function in order to not pass the whole object, otherwise it wouldn’t serialize. It’s really just a simple helper taking the properties of interest for us (and letting away all the rest).</p> | |||
| <p>Another peer can observe the changes, like this:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">handle</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s2">"change"</span><span class="p">,</span><span class="w"> </span><span class="p">({</span><span class="w"> </span><span class="nx">doc</span><span class="p">,</span><span class="w"> </span><span class="nx">patches</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">patches</span><span class="p">.</span><span class="nx">forEach</span><span class="p">(({</span><span class="w"> </span><span class="nx">action</span><span class="p">,</span><span class="w"> </span><span class="nx">path</span><span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">path</span><span class="p">.</span><span class="nx">length</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mf">2</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nx">action</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"insert"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">doc</span><span class="p">[</span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">]];</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">path</span><span class="p">[</span><span class="mf">0</span><span class="p">],</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>The “patch” interface is a bit less verbose than the one from Y.js. It wasn’t well documented, so I had to play around with the messages to understand exactly what the possible actions were. In the end, it gets me what I’m looking for, the changes that occurred to my data.</p> | |||
| <p>Here I’m using <code>path.length == 2 && action === 'insert'</code> to detect that a change occurred to the marker. Since we don’t make a difference between the creation of a marker and its update (when being moved), it works well for us.</p> | |||
| <h4 id="python-bindings_1">Python bindings</h4> | |||
| <p>There are <a href="https://github.com/automerge/automerge-py">python bindings</a> for automerge “core”. (It doesn’t — yet — provide ways to interact with “repos”).</p> | |||
| <h4 id="library-size_1">Library size</h4> | |||
| <ul> | |||
| <li>Size of the automerge + automerge repo: 1,64 mb</li> | |||
| <li>Size of the WebSocket provider: 0,10 mb </li> | |||
| </ul> | |||
| <p>This is quite a large bundle size, and the team behind automerge is aware of it and working on a solution.</p> | |||
| <h4 id="the-data-being-transmitted_1">The data being transmitted</h4> | |||
| <p>In the same scenario, adding 20 points on one client, then 20 points in another generates 90 messages and 24,94 Ko of data transmitted (~12 Ko sent and ~12Ko received), so approximately 75 bytes per edit.</p> | |||
| <p>Pros:</p> | |||
| <ul> | |||
| <li>There is an <span class="caps">API</span> to <a href="https://automerge.org/docs/documents/conflicts/">get informed when a conflict occurred</a></li> | |||
| <li>Python bindings are currently being worked on, soon to reach a stable version</li> | |||
| <li>The team was overall very responsive and trying to help.</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>The JavaScript is currently generated via Web Assembly, which could make it harder to debug.</li> | |||
| <li>The large bundle size of the generated files.</li> | |||
| </ul> | |||
| <hr> | |||
| <h3 id="json-joy"><span class="caps">JSON</span> Joy</h3> | |||
| <p><a href="https://jsonjoy.com"><span class="caps">JSON</span> Joy</a> is the latest to the party.</p> | |||
| <p>It takes another stake by providing multiple libraries with a small functional perimeter. It sounds promising, even if still quite new, and would leave us with the hands free in order to implement the protocol that would work for us.</p> | |||
| <p>Here is how to use it. On the different peers you start with different forks of the same document:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="nx">Model</span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s1">'json-joy/es2020/json-crdt'</span><span class="p">;</span> | |||
| <span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">s</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch"</span><span class="p">;</span> | |||
| <span class="c1">// Initiate a model with a custom ID</span> | |||
| <span class="kd">const</span><span class="w"> </span><span class="nx">rootModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">Model</span><span class="p">.</span><span class="nx">withLogicalClock</span><span class="p">(</span><span class="mf">11111111</span><span class="p">);</span> | |||
| <span class="c1">// populate it with default data</span> | |||
| <span class="nx">rootModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">root</span><span class="p">({</span> | |||
| <span class="w"> </span><span class="nx">markers</span><span class="o">:</span><span class="w"> </span><span class="p">{},</span> | |||
| <span class="p">});</span> | |||
| <span class="c1">// Fork it on each client</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">userModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">rootModel</span><span class="p">.</span><span class="nx">fork</span><span class="p">();</span> | |||
| </code></pre></div> | |||
| <p>When adding a new marker, we can define a new constant, by using <code>s.con</code>…</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">obj</span><span class="p">([</span><span class="s2">"markers"</span><span class="p">]).</span><span class="nx">set</span><span class="p">({</span> | |||
| <span class="w"> </span><span class="p">[</span><span class="nx">uuid</span><span class="p">]</span><span class="o">:</span><span class="w"> </span><span class="nx">s</span><span class="p">.</span><span class="nx">con</span><span class="p">(</span><span class="nx">target</span><span class="p">.</span><span class="nx">_latlng</span><span class="p">),</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>… and then create a patch and send it to the other peers:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="k">import</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nx">encode</span><span class="p">,</span><span class="w"> </span><span class="nx">decode</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="kr">from</span><span class="w"> </span><span class="s2">"json-joy/es6/json-crdt-patch/codec/verbose"</span><span class="p">;</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">flush</span><span class="p">();</span> | |||
| <span class="kd">let</span><span class="w"> </span><span class="nx">payload</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">encode</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>On the other peers, when we receive a patch message, decode it and apply it:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="nx">patch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">decode</span><span class="p">(</span><span class="nx">payload</span><span class="p">);</span> | |||
| <span class="nx">model</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">patch</span><span class="p">);</span> | |||
| </code></pre></div> | |||
| <p>We can observe the changes this way:</p> | |||
| <div class="highlight"><pre><span></span><code><span class="nx">userModel</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">onPatch</span><span class="p">.</span><span class="nx">listen</span><span class="p">((</span><span class="nx">patch</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="nx">patch</span><span class="p">.</span><span class="nx">ops</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">op</span><span class="p">)</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nx">op</span><span class="p">.</span><span class="nx">name</span><span class="p">()</span><span class="w"> </span><span class="o">===</span><span class="w"> </span><span class="s2">"ins_obj"</span><span class="p">)</span><span class="w"> </span><span class="p">{</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">op</span><span class="p">.</span><span class="nx">data</span><span class="p">[</span><span class="mf">0</span><span class="p">][</span><span class="mf">0</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">value</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">userModel</span><span class="p">.</span><span class="nx">view</span><span class="p">().</span><span class="nx">markers</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span> | |||
| <span class="w"> </span><span class="nx">upsertMarker</span><span class="p">({</span><span class="w"> </span><span class="nx">latlng</span><span class="o">:</span><span class="w"> </span><span class="nx">value</span><span class="p">,</span><span class="w"> </span><span class="nx">uuid</span><span class="o">:</span><span class="w"> </span><span class="nx">key</span><span class="p">,</span><span class="w"> </span><span class="nx">local</span><span class="o">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">});</span> | |||
| <span class="w"> </span><span class="p">}</span> | |||
| <span class="w"> </span><span class="p">});</span> | |||
| <span class="p">});</span> | |||
| </code></pre></div> | |||
| <p>Similarly to what we did with automerge, we’re having a look into the patch, and filter on the operations of interest for us (<code>ins_obj</code>). The names of the operations aren’t clearly specified by the spec.</p> | |||
| <p>Metrics:</p> | |||
| <ul> | |||
| <li>Size: 143 ko</li> | |||
| <li>Data transmitted for 2 peers and 40 edits: (35 bytes per edit)</li> | |||
| </ul> | |||
| <p>Pros:</p> | |||
| <ul> | |||
| <li>Small atomic libraries, making it easy to use only the parts we need.</li> | |||
| <li>The interface <a href="https://jsonjoy.com/libs/json-joy-js/json-crdt/guide/node-types">proposes to store different type of data</a> (constants, values, arrays, etc.)</li> | |||
| <li>Distributed as different type of <span class="caps">JS</span> bundles (modules, wasm, etc.)</li> | |||
| <li>Low level, so you know what you’re doing</li> | |||
| </ul> | |||
| <p>Cons:</p> | |||
| <ul> | |||
| <li>It doesn’t provide a high level interface for sync</li> | |||
| <li>It’s currently a one-person project, without clear community channels to gather with other interested folks.</li> | |||
| <li>Quite recent, so probably rough spots are to be found</li> | |||
| </ul> | |||
| <hr> | |||
| <h2 id="part-3-comparison-table">Part 3: Comparison table</h2> | |||
| <p>I found Y.js and Automerge quite similar for my use case, while <span class="caps">JSON</span> Joy was taking a different (less “all-included”) approach. Here is a summary table to help read the differences I found.</p> | |||
| <table> | |||
| <thead> | |||
| <tr> | |||
| <th></th> | |||
| <th>Y.js</th> | |||
| <th>Automerge</th> | |||
| <th><span class="caps">JSON</span> Joy</th> | |||
| </tr> | |||
| </thead> | |||
| <tbody> | |||
| <tr> | |||
| <td>Python bindings</td> | |||
| <td><a href="https://github.com/y-crdt/ypy">Yes</a></td> | |||
| <td><a href="https://github.com/automerge/automerge-py">Yes</a></td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Syncing</td> | |||
| <td>Native <span class="caps">JS</span> structures</td> | |||
| <td>Transactional functions</td> | |||
| <td>Specific types (bonus points for handling constants)</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Coded in</td> | |||
| <td>JavaScript / Rust</td> | |||
| <td>TypeScript / Rust</td> | |||
| <td>Typescript</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Awareness protocol</td> | |||
| <td>Yes, with presence</td> | |||
| <td>Yes, without presence</td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Conflict-detection <span class="caps">API</span></td> | |||
| <td>No</td> | |||
| <td>Yes</td> | |||
| <td>No</td> | |||
| </tr> | |||
| <tr> | |||
| <td>Library size</td> | |||
| <td><strong>24.3Ko</strong> §<br></td> | |||
| <td><strong>1,74 mb</strong> §</td> | |||
| <td><strong>143 ko</strong></td> | |||
| </tr> | |||
| </tbody> | |||
| </table> | |||
| <p>§ size of the connectors included</p> | |||
| <h3 id="working-with-patches">Working with patches</h3> | |||
| <p>In order to observe the changes, we need to inspect the given patches and work on what we find. I found the different libraries expose different sets of APIs. All of these APIs were quite a bit hard to find, and it’s not clear if they are public or private.</p> | |||
| <p>One thing to keep in mind is that these “patch” events happen only once per patch received. You can see it as a “diff” of the state between the current and the incoming states.</p> | |||
| <ul> | |||
| <li>Y.js exposes a utility which is able to tell you what the action on the key is (“delete”, “update” and “add”)</li> | |||
| <li>Automerge and <span class="caps">JSON</span> Joy, on the other hand, don’t provide such utility functions, meaning you would need to find that out yourself.</li> | |||
| </ul> | |||
| <h3 id="conclusion">Conclusion</h3> | |||
| <p>The goal here is not to tell which one of these libraries is the best one. They’re all great and have their strenghs. None of them implement the high-level <span class="caps">API</span> I was expecting, where the clients talk with each other and the server just relays messages, but maybe it’s because it’s better in general to have the server have the representation of the data, saving a roundtrip for the clients.</p> | |||
| <p>I wasn’t expecting to have a look at patches to understand what changed at the low level. The way it’s currently implemented is very suitable for “reactive” applications, but require more involvement to sync between the CRDTs state and the application state.</p> | |||
| <p>In the end, adding CRDTs is made very simple, probably due to the fact all we really need is a sort of distributed key/value store.</p> | |||
| <p>I’m not sure yet which library we will end up using for uMap (if any), but my understanding is clearer than it was when I started. I guess that’s what progress looks like 😎</p> | |||
| <h3 id="yata-and-rga"><span class="caps">YATA</span> and <span class="caps">RGA</span></h3> | |||
| <p>While researching, I found that the two popular CRDTs implementation out there use different approaches for the virtual counter:</p> | |||
| <blockquote> | |||
| <ul> | |||
| <li><strong><span class="caps">RGA</span></strong> [used by Automerge] maintains a single globally incremented counter (which can be ordinary integer value), that’s updated anytime we detect that remote insert has an id | |||
| with sequence number higher that local counter. Therefore, every time we produce | |||
| a new insert operation, we give it a highest counter value known at the time.</li> | |||
| <li><strong><span class="caps">YATA</span></strong> [used by Yjs] also uses a single integer value, however unlike in case of <span class="caps">RGA</span> we | |||
| don’t use a single counter shared with other replicas, but rather let each | |||
| peer keep its own, which is incremented monotonically only by that peer. Since | |||
| increments are monotonic, we can also use them to detect missing operations eg. | |||
| updates marked as A:1 and A:3 imply, that there must be another (potentially | |||
| missing) update A:2.</li> | |||
| </ul> | |||
| </blockquote> | |||
| <h3 id="resources">Resources</h3> | |||
| <ul> | |||
| <li><a href="https://www.youtube.com/watch?v=x7drE24geUw">CRDTs: The Hard Parts</a>, a video by Martin Kleppmann where he explains the current state of the art of CRDTs, and why some problems aren’t solved yet.</li> | |||
| <li><a href="https://jakelazaroff.com/words/an-interactive-intro-to-crdts/">An Interactive Intro to CRDTs</a> gets you trough different steps to understand what are CRDTs, and how to implement a <span class="caps">LWW</span> Register.</li> | |||
| <li><a href="https://www.bartoszsypytkowski.com/the-state-of-a-state-based-crdts/">Bartosz Sypytkowski</a> introduction on CRDTs, with practical examples is very intuitive.</li> | |||
| <li><a href="https://jzhao.xyz/thoughts/CRDT-Implementations"><span class="caps">CRDT</span> Implementations</a> are notes by Jacky which were useful to me when understanding CRDTs.</li> | |||
| </ul> | |||
+ 224
- 0
cache/2024/46dc6f44f3e34c4c0626ad4b13dba768/index.html
View File
| @@ -0,0 +1,224 @@ | |||
| <!doctype html><!-- This is a valid HTML5 document. --> | |||
| <!-- Screen readers, SEO, extensions and so on. --> | |||
| <html lang="en"> | |||
| <!-- Has to be within the first 1024 bytes, hence before the `title` element | |||
| See: https://www.w3.org/TR/2012/CR-html5-20121217/document-metadata.html#charset --> | |||
| <meta charset="utf-8"> | |||
| <!-- Why no `X-UA-Compatible` meta: https://stackoverflow.com/a/6771584 --> | |||
| <!-- The viewport meta is quite crowded and we are responsible for that. | |||
| See: https://codepen.io/tigt/post/meta-viewport-for-2015 --> | |||
| <meta name="viewport" content="width=device-width,initial-scale=1"> | |||
| <!-- Required to make a valid HTML5 document. --> | |||
| <title>command center: Prints (archive) — David Larlet</title> | |||
| <meta name="description" content="Publication mise en cache pour en conserver une trace."> | |||
| <!-- That good ol' feed, subscribe :). --> | |||
| <link rel="alternate" type="application/atom+xml" title="Feed" href="/david/log/"> | |||
| <!-- Generated from https://realfavicongenerator.net/ such a mess. --> | |||
| <link rel="apple-touch-icon" sizes="180x180" href="/static/david/icons2/apple-touch-icon.png"> | |||
| <link rel="icon" type="image/png" sizes="32x32" href="/static/david/icons2/favicon-32x32.png"> | |||
| <link rel="icon" type="image/png" sizes="16x16" href="/static/david/icons2/favicon-16x16.png"> | |||
| <link rel="manifest" href="/static/david/icons2/site.webmanifest"> | |||
| <link rel="mask-icon" href="/static/david/icons2/safari-pinned-tab.svg" color="#07486c"> | |||
| <link rel="shortcut icon" href="/static/david/icons2/favicon.ico"> | |||
| <meta name="msapplication-TileColor" content="#f7f7f7"> | |||
| <meta name="msapplication-config" content="/static/david/icons2/browserconfig.xml"> | |||
| <meta name="theme-color" content="#f7f7f7" media="(prefers-color-scheme: light)"> | |||
| <meta name="theme-color" content="#272727" media="(prefers-color-scheme: dark)"> | |||
| <!-- Is that even respected? Retrospectively? What a shAItshow… | |||
| https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ --> | |||
| <meta name="robots" content="noai, noimageai"> | |||
| <!-- Documented, feel free to shoot an email. --> | |||
| <link rel="stylesheet" href="/static/david/css/style_2021-01-20.css"> | |||
| <!-- See https://www.zachleat.com/web/comprehensive-webfonts/ for the trade-off. --> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t4_poly_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: light), (prefers-color-scheme: no-preference)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_regular.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_bold.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <link rel="preload" href="/static/david/css/fonts/triplicate_t3_italic.woff2" as="font" type="font/woff2" media="(prefers-color-scheme: dark)" crossorigin> | |||
| <script> | |||
| function toggleTheme(themeName) { | |||
| document.documentElement.classList.toggle( | |||
| 'forced-dark', | |||
| themeName === 'dark' | |||
| ) | |||
| document.documentElement.classList.toggle( | |||
| 'forced-light', | |||
| themeName === 'light' | |||
| ) | |||
| } | |||
| const selectedTheme = localStorage.getItem('theme') | |||
| if (selectedTheme !== 'undefined') { | |||
| toggleTheme(selectedTheme) | |||
| } | |||
| </script> | |||
| <meta name="robots" content="noindex, nofollow"> | |||
| <meta content="origin-when-cross-origin" name="referrer"> | |||
| <!-- Canonical URL for SEO purposes --> | |||
| <link rel="canonical" href="https://commandcenter.blogspot.com/2014/08/prints.html"> | |||
| <body class="remarkdown h1-underline h2-underline h3-underline em-underscore hr-center ul-star pre-tick" data-instant-intensity="viewport-all"> | |||
| <article> | |||
| <header> | |||
| <h1>command center: Prints</h1> | |||
| </header> | |||
| <nav> | |||
| <p class="center"> | |||
| <a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use> | |||
| </svg> Accueil</a> • | |||
| <a href="https://commandcenter.blogspot.com/2014/08/prints.html" title="Lien vers le contenu original">Source originale</a> | |||
| <br> | |||
| Mis en cache le 2024-03-25 | |||
| </p> | |||
| </nav> | |||
| <hr> | |||
| <p> | |||
| Two long-buried caches of photographs came to light last year. One was a stack of cellulose nitrate negatives made on the Scott Antarctic expedition almost a hundred years ago. Over time, they became stuck together into a moldy brick, but it was possible to tease the negatives apart and see what they revealed. You can view the images at the web site of the </p> | |||
| <p><a href="http://www.nzaht.org/AHT/antarctic-photos/" rel="nofollow" target="_blank">New Zealand Antarctic Heritage Trust</a> | |||
| <p>. The results show ragged edges and mold spots but, even beyond their historical importance, the photographs are evocative and in some cases very beautiful. | |||
| </p></p> | |||
| <p> | |||
| The other cache contained images not quite so old and of less general interest but of personal importance. My mother moved from the house she had occupied for decades into a smaller apartment and while preparing to move she found the proverbial shoe box of old pictures in a closet. Some of the images are from my youth, some from hers, and some even from her parents'. One of the photographs, from 1931, shows my paternal great-grandparents. I never met my paternal grandparents, let alone great-grandparents, so this photograph touches something almost primordial for me. And some of the photographs in the box were even older. | |||
| </p> | |||
| <p> | |||
| Due to the miracle of photography, we are able to see over a hundred years into the past. Of course this is not news; all of us have seen 19th century photographs by the pioneers of the medium. By the turn of the 20th century photography was so common that huge numbers of images, from the historical to the mundane, had been created. And sometimes we are lucky enough to chance upon forgotten images that open a window into a past that would otherwise fade from view. | |||
| </p> | |||
| <p> | |||
| But such windows are becoming rare. A hundred years from now, there will be far fewer photo caches to find. Although the transition to digital photography has made photos almost unimaginably commonplace—one estimate puts the number of shutter activations at a trillion images worldwide per year—very few of those images become artifacts that can be left in a shoe box. | |||
| </p> | |||
| <p> | |||
| We live in what has been named a </p> | |||
| <p><a href="http://en.wikipedia.org/wiki/Digital_dark_age" rel="nofollow" target="_blank">Digital Dark Age</a> | |||
| <p>. Because digital technology evolves so fast, we are rapidly losing the ability to understand yesterday's media. As file formats change, software becomes obsolete, and hardware becomes outmoded, old digital files become unreadable and unrecoverable. | |||
| </p></p> | |||
| <p> | |||
| There are many examples of lost information, but here is an illustrative story of disaster narrowly averted. Early development of the Unix operating system, which became the software foundation for the Internet, was done in the late 1960s and early 1970s on Digital Equipment Corporation computers. Backups were made on a magnetic medium called a </p> | |||
| <p><a href="http://en.wikipedia.org/wiki/DECtape" rel="nofollow" target="_blank">DECtape</a> | |||
| <p>. By the mid 1970s, DECtape was obsolete and by the 1980s there were no remaining DECtape drives that could read the old backups. The scientists in the original Unix lab had kept a box of old backups under the raised floor of the computer room, but the tapes had spontaneously become unreadable because the device to read them no longer existed in the lab or anywhere else as far as anyone knew. And even if it did, no computer that could run the device was still powered on. Fortunately, around 1990 Paul Vixie and Keith Bostic, working for a different company, stumbled across an old junked DECtape drive and managed to get it up and running again by resurrecting an old computer to connect it to. They contacted the Unix research group and offered one last chance to recover the data on the backup tapes before the computer and DECtape drive were finally decommissioned. Time and resources were limited, but some of the key archival pieces of early Unix development were recovered through this combination of charity and a great deal of luck. This story has a happy ending, but not all digital archives survive. Far from it. | |||
| </p></p> | |||
| <p> | |||
| The problem is that as technology advances, data needs to be curated. Files need to have their formats converted, and then transferred to new media. A backup disk in a box somewhere might be unreadable a few years from now. Its format may be obsolete, the software to read it might not run on current hardware, or the media might have physically decayed. NASA lost a large part of the data collected by the Viking Mars missions because the iron oxide fell off the tapes storing the data. | |||
| </p> | |||
| <p> | |||
| Backups are important but they too are temporary, subject to the same problems as the data they attempt to protect. Backup software can become obsolete and media can fail. The same affliction that damaged the Viking tapes also wiped out my personal backup archive; I lost the only copy of my computer work from the 1970s. (It's worth noting my negatives and prints from the period survived.) | |||
| </p> | |||
| <p> | |||
| It's not just tapes that go bad. Consider CDs and DVDs, media often used for backup. The disks, especially the writable kind use for backups, are very fragile, much more so than the mass-produced read-only kind used to store music and movies. Within a few years, especially in humid environments, the metal film can separate from the backing medium. Even if the backup medium survives, the formats used to store the backups might become obsolete. The software that reads the backups might not run on the next computer one buys. Today, CDs are already becoming relics; many computers today do not even come with a CD or DVD drive. What were once the gold standard for backup are already looking old-fashioned just a few years on. They will be antiquated and obscure a century from now. | |||
| </p> | |||
| <p> | |||
| To summarize, digital information requires maintenance. It's not sufficient to make backups; the backups also need to be maintained, upgraded, transferred, and curated. Without conscientious care, the data of today will be lost forever in a few years. Even with care, it's possible through software or hardware changes to lose access forever. That shoebox of old backup CDs will be unreadable soon. | |||
| </p> | |||
| <p> | |||
| Which brings us back to those old photo caches. They held negatives and prints, physical objects that stored images. They needed no attention, no curating, no updating. They sat untended and forgotten for decades, but through all that time faithfully held their information, waiting for a future discoverer. As a result, we can all see what the Scott Antarctic expedition saw, and I can see what my great-grandparents looked like. | |||
| </p> | |||
| <p> | |||
| It is a sad irony that modern technology makes it unlikely that future generations will see the images made today. | |||
| </p> | |||
| <p> | |||
| Ask yourself whether your great-grandchildren will be able to see your photographs. If the images exist only as a digital image file, the answer is almost certainly, "No". If, however, there are physical prints, the odds improve. Those digital images need to be made real to endure. Without a print, a digital photograph has no future. | |||
| </p> | |||
| <p> | |||
| We live in a Digital Dark Age, but as individuals we can shine a little light. If you are one of the uncounted photographers who enjoy digital photography, keep in mind the fragility of data. When you have a digital image you care about, for whatever reason, artistic or sentimental, please make a print and put that print away. It will sit quietly in the dark, holding fast, never forgetting, ready to reveal itself to a grateful future generation. | |||
| </p> | |||
| </article> | |||
| <hr> | |||
| <footer> | |||
| <p> | |||
| <a href="/david/" title="Aller à l’accueil"><svg class="icon icon-home"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-home"></use> | |||
| </svg> Accueil</a> • | |||
| <a href="/david/log/" title="Accès au flux RSS"><svg class="icon icon-rss2"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-rss2"></use> | |||
| </svg> Suivre</a> • | |||
| <a href="http://larlet.com" title="Go to my English profile" data-instant><svg class="icon icon-user-tie"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-user-tie"></use> | |||
| </svg> Pro</a> • | |||
| <a href="mailto:david%40larlet.fr" title="Envoyer un courriel"><svg class="icon icon-mail"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-mail"></use> | |||
| </svg> Email</a> • | |||
| <abbr class="nowrap" title="Hébergeur : Alwaysdata, 62 rue Tiquetonne 75002 Paris, +33184162340"><svg class="icon icon-hammer2"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-hammer2"></use> | |||
| </svg> Légal</abbr> | |||
| </p> | |||
| <template id="theme-selector"> | |||
| <form> | |||
| <fieldset> | |||
| <legend><svg class="icon icon-brightness-contrast"> | |||
| <use xlink:href="/static/david/icons2/symbol-defs-2021-12.svg#icon-brightness-contrast"></use> | |||
| </svg> Thème</legend> | |||
| <label> | |||
| <input type="radio" value="auto" name="chosen-color-scheme" checked> Auto | |||
| </label> | |||
| <label> | |||
| <input type="radio" value="dark" name="chosen-color-scheme"> Foncé | |||
| </label> | |||
| <label> | |||
| <input type="radio" value="light" name="chosen-color-scheme"> Clair | |||
| </label> | |||
| </fieldset> | |||
| </form> | |||
| </template> | |||
| </footer> | |||
| <script src="/static/david/js/instantpage-5.1.0.min.js" type="module"></script> | |||
| <script> | |||
| function loadThemeForm(templateName) { | |||
| const themeSelectorTemplate = document.querySelector(templateName) | |||
| const form = themeSelectorTemplate.content.firstElementChild | |||
| themeSelectorTemplate.replaceWith(form) | |||
| form.addEventListener('change', (e) => { | |||
| const chosenColorScheme = e.target.value | |||
| localStorage.setItem('theme', chosenColorScheme) | |||
| toggleTheme(chosenColorScheme) | |||
| }) | |||
| const selectedTheme = localStorage.getItem('theme') | |||
| if (selectedTheme && selectedTheme !== 'undefined') { | |||
| form.querySelector(`[value="${selectedTheme}"]`).checked = true | |||
| } | |||
| } | |||
| const prefersColorSchemeDark = '(prefers-color-scheme: dark)' | |||
| window.addEventListener('load', () => { | |||
| let hasDarkRules = false | |||
| for (const styleSheet of Array.from(document.styleSheets)) { | |||
| let mediaRules = [] | |||
| for (const cssRule of styleSheet.cssRules) { | |||
| if (cssRule.type !== CSSRule.MEDIA_RULE) { | |||
| continue | |||
| } | |||
| // WARNING: Safari does not have/supports `conditionText`. | |||
| if (cssRule.conditionText) { | |||
| if (cssRule.conditionText !== prefersColorSchemeDark) { | |||
| continue | |||
| } | |||
| } else { | |||
| if (cssRule.cssText.startsWith(prefersColorSchemeDark)) { | |||
| continue | |||
| } | |||
| } | |||
| mediaRules = mediaRules.concat(Array.from(cssRule.cssRules)) | |||
| } | |||
| // WARNING: do not try to insert a Rule to a styleSheet you are | |||
| // currently iterating on, otherwise the browser will be stuck | |||
| // in a infinite loop… | |||
| for (const mediaRule of mediaRules) { | |||
| styleSheet.insertRule(mediaRule.cssText) | |||
| hasDarkRules = true | |||
| } | |||
| } | |||
| if (hasDarkRules) { | |||
| loadThemeForm('#theme-selector') | |||
| } | |||
| }) | |||
| </script> | |||
| </body> | |||
| </html> | |||
+ 38
- 0
cache/2024/46dc6f44f3e34c4c0626ad4b13dba768/index.md
View File
| @@ -0,0 +1,38 @@ | |||
| title: command center: Prints | |||
| url: https://commandcenter.blogspot.com/2014/08/prints.html | |||
| hash_url: 46dc6f44f3e34c4c0626ad4b13dba768 | |||
| archive_date: 2024-03-25 | |||
| og_image: https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQvTZvSIAYokvx00Scs3gCg8rR2X74iruzto6wDM1Tth-ZeUPrEH01XM3NPwhLa62ga6pQlMMMzKXrsh3rP_BzOSL4eahGtRZw1LaESnDhmHFmot1OcEqsKsMr84KED9HO4m2F7VJcksZrE-U7WtacxMmXeRTeaveAIEvmHwvs36TOfSnbizPzzw/w400-h295/golang.org-2009.png | |||
| description: Two long-buried caches of photographs came to light last year. One was a stack of cellulose nitrate negatives made on the Scott Antarctic ex... | |||
| favicon: https://commandcenter.blogspot.com/favicon.ico | |||
| language: en_US | |||
| <p> | |||
| Two long-buried caches of photographs came to light last year. One was a stack of cellulose nitrate negatives made on the Scott Antarctic expedition almost a hundred years ago. Over time, they became stuck together into a moldy brick, but it was possible to tease the negatives apart and see what they revealed. You can view the images at the web site of the </p><a href="http://www.nzaht.org/AHT/antarctic-photos/" rel="nofollow" target="_blank">New Zealand Antarctic Heritage Trust</a><p>. The results show ragged edges and mold spots but, even beyond their historical importance, the photographs are evocative and in some cases very beautiful. | |||
| </p><p> | |||
| The other cache contained images not quite so old and of less general interest but of personal importance. My mother moved from the house she had occupied for decades into a smaller apartment and while preparing to move she found the proverbial shoe box of old pictures in a closet. Some of the images are from my youth, some from hers, and some even from her parents'. One of the photographs, from 1931, shows my paternal great-grandparents. I never met my paternal grandparents, let alone great-grandparents, so this photograph touches something almost primordial for me. And some of the photographs in the box were even older. | |||
| </p><p> | |||
| Due to the miracle of photography, we are able to see over a hundred years into the past. Of course this is not news; all of us have seen 19th century photographs by the pioneers of the medium. By the turn of the 20th century photography was so common that huge numbers of images, from the historical to the mundane, had been created. And sometimes we are lucky enough to chance upon forgotten images that open a window into a past that would otherwise fade from view. | |||
| </p><p> | |||
| But such windows are becoming rare. A hundred years from now, there will be far fewer photo caches to find. Although the transition to digital photography has made photos almost unimaginably commonplace—one estimate puts the number of shutter activations at a trillion images worldwide per year—very few of those images become artifacts that can be left in a shoe box. | |||
| </p><p> | |||
| We live in what has been named a </p><a href="http://en.wikipedia.org/wiki/Digital_dark_age" rel="nofollow" target="_blank">Digital Dark Age</a><p>. Because digital technology evolves so fast, we are rapidly losing the ability to understand yesterday's media. As file formats change, software becomes obsolete, and hardware becomes outmoded, old digital files become unreadable and unrecoverable. | |||
| </p><p> | |||
| There are many examples of lost information, but here is an illustrative story of disaster narrowly averted. Early development of the Unix operating system, which became the software foundation for the Internet, was done in the late 1960s and early 1970s on Digital Equipment Corporation computers. Backups were made on a magnetic medium called a </p><a href="http://en.wikipedia.org/wiki/DECtape" rel="nofollow" target="_blank">DECtape</a><p>. By the mid 1970s, DECtape was obsolete and by the 1980s there were no remaining DECtape drives that could read the old backups. The scientists in the original Unix lab had kept a box of old backups under the raised floor of the computer room, but the tapes had spontaneously become unreadable because the device to read them no longer existed in the lab or anywhere else as far as anyone knew. And even if it did, no computer that could run the device was still powered on. Fortunately, around 1990 Paul Vixie and Keith Bostic, working for a different company, stumbled across an old junked DECtape drive and managed to get it up and running again by resurrecting an old computer to connect it to. They contacted the Unix research group and offered one last chance to recover the data on the backup tapes before the computer and DECtape drive were finally decommissioned. Time and resources were limited, but some of the key archival pieces of early Unix development were recovered through this combination of charity and a great deal of luck. This story has a happy ending, but not all digital archives survive. Far from it. | |||
| </p><p> | |||
| The problem is that as technology advances, data needs to be curated. Files need to have their formats converted, and then transferred to new media. A backup disk in a box somewhere might be unreadable a few years from now. Its format may be obsolete, the software to read it might not run on current hardware, or the media might have physically decayed. NASA lost a large part of the data collected by the Viking Mars missions because the iron oxide fell off the tapes storing the data. | |||
| </p><p> | |||
| Backups are important but they too are temporary, subject to the same problems as the data they attempt to protect. Backup software can become obsolete and media can fail. The same affliction that damaged the Viking tapes also wiped out my personal backup archive; I lost the only copy of my computer work from the 1970s. (It's worth noting my negatives and prints from the period survived.) | |||
| </p><p> | |||
| It's not just tapes that go bad. Consider CDs and DVDs, media often used for backup. The disks, especially the writable kind use for backups, are very fragile, much more so than the mass-produced read-only kind used to store music and movies. Within a few years, especially in humid environments, the metal film can separate from the backing medium. Even if the backup medium survives, the formats used to store the backups might become obsolete. The software that reads the backups might not run on the next computer one buys. Today, CDs are already becoming relics; many computers today do not even come with a CD or DVD drive. What were once the gold standard for backup are already looking old-fashioned just a few years on. They will be antiquated and obscure a century from now. | |||
| </p><p> | |||
| To summarize, digital information requires maintenance. It's not sufficient to make backups; the backups also need to be maintained, upgraded, transferred, and curated. Without conscientious care, the data of today will be lost forever in a few years. Even with care, it's possible through software or hardware changes to lose access forever. That shoebox of old backup CDs will be unreadable soon. | |||
| </p><p> | |||
| Which brings us back to those old photo caches. They held negatives and prints, physical objects that stored images. They needed no attention, no curating, no updating. They sat untended and forgotten for decades, but through all that time faithfully held their information, waiting for a future discoverer. As a result, we can all see what the Scott Antarctic expedition saw, and I can see what my great-grandparents looked like. | |||
| </p><p> | |||
| It is a sad irony that modern technology makes it unlikely that future generations will see the images made today. | |||
| </p><p> | |||
| Ask yourself whether your great-grandchildren will be able to see your photographs. If the images exist only as a digital image file, the answer is almost certainly, "No". If, however, there are physical prints, the odds improve. Those digital images need to be made real to endure. Without a print, a digital photograph has no future. | |||
| </p><p> | |||
| We live in a Digital Dark Age, but as individuals we can shine a little light. If you are one of the uncounted photographers who enjoy digital photography, keep in mind the fragility of data. When you have a digital image you care about, for whatever reason, artistic or sentimental, please make a print and put that print away. It will sit quietly in the dark, holding fast, never forgetting, ready to reveal itself to a grateful future generation. | |||
| </p> | |||
+ 4
- 0
cache/2024/index.html
View File
| @@ -168,6 +168,8 @@ | |||
| <li><a href="/david/cache/2024/956819385548bba6e768563b12edc2d6/" title="Accès à l’article dans le cache local : herbe">herbe</a> (<a href="https://www.la-grange.net/2024/01/24/herbe" title="Accès à l’article original distant : herbe">original</a>)</li> | |||
| <li><a href="/david/cache/2024/46dc6f44f3e34c4c0626ad4b13dba768/" title="Accès à l’article dans le cache local : command center: Prints">command center: Prints</a> (<a href="https://commandcenter.blogspot.com/2014/08/prints.html" title="Accès à l’article original distant : command center: Prints">original</a>)</li> | |||
| <li><a href="/david/cache/2024/a988555163e09729b925dbf715ce256c/" title="Accès à l’article dans le cache local : How web bloat impacts users with slow devices">How web bloat impacts users with slow devices</a> (<a href="https://danluu.com/slow-device/" title="Accès à l’article original distant : How web bloat impacts users with slow devices">original</a>)</li> | |||
| <li><a href="/david/cache/2024/2a1235215c277ebb8a0e9acb7ffd91e0/" title="Accès à l’article dans le cache local : drab - A Headless Custom Element Library">drab - A Headless Custom Element Library</a> (<a href="https://drab.robino.dev/" title="Accès à l’article original distant : drab - A Headless Custom Element Library">original</a>)</li> | |||
| @@ -260,6 +262,8 @@ | |||
| <li><a href="/david/cache/2024/4c8a04c4c0e928bd78f22db77425bb47/" title="Accès à l’article dans le cache local : No more forever projects">No more forever projects</a> (<a href="https://dianaberlin.com/posts/no-more-forever-projects" title="Accès à l’article original distant : No more forever projects">original</a>)</li> | |||
| <li><a href="/david/cache/2024/2c0b2588dfcd3a194da4133c7505cd3e/" title="Accès à l’article dans le cache local : A comparison of JavaScript CRDTs">A comparison of JavaScript CRDTs</a> (<a href="https://blog.notmyidea.org/a-comparison-of-javascript-crdts.html" title="Accès à l’article original distant : A comparison of JavaScript CRDTs">original</a>)</li> | |||
| <li><a href="/david/cache/2024/ea2cfc9aa425a6967d2cacd9f96ceb9e/" title="Accès à l’article dans le cache local : Ask LukeW: New Ways into Web Content">Ask LukeW: New Ways into Web Content</a> (<a href="https://lukew.com/ff/entry.asp?2008" title="Accès à l’article original distant : Ask LukeW: New Ways into Web Content">original</a>)</li> | |||
| <li><a href="/david/cache/2024/f5294ac20ea593cce56caf2379813a4a/" title="Accès à l’article dans le cache local : Chiroto T. Datoca">Chiroto T. Datoca</a> (<a href="https://www.hypothermia.fr/2024/03/chiroto-t-datoca/" title="Accès à l’article original distant : Chiroto T. Datoca">original</a>)</li> | |||
Loading…