larlet-fr-david-cache/cache/2023/49f2ce04dd0beb94dc2f662163bc6339/index.md

title: Some notes on Local-First Development
url: https://bricolage.io/some-notes-on-local-first-development/
hash_url: 49f2ce04dd0beb94dc2f662163bc6339

<p>A few months ago in June, I attended <a href="https://www.youtube.com/results?search_query=Local-First+Meetup+Berlin+%231">a local-first meetup in Berlin</a> organized by Johannes Schickling, formerly the founder of Prisma. An intellectual crackle filled the air as we watched demos of new libraries and products. Many of us had been independently playing around with local-first development ideas for a while — in my case, over a decade — and the in-person meetup gave us the chance to trade notes late into the night.</p>
<p><span class="gatsby-resp-image-wrapper">
      <a class="gatsby-resp-image-link" href="https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/49ee4/berlin-meetup.jpg" target="_blank" rel="noopener">
    <span class="gatsby-resp-image-background-image"></span>
  <img class="gatsby-resp-image-image" alt="Picture of Berlin Local-First meetup" title="" src="https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/1c72d/berlin-meetup.jpg" srcset="https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/a80bd/berlin-meetup.jpg 148w,
https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/1c91a/berlin-meetup.jpg 295w,
https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/1c72d/berlin-meetup.jpg 590w,
https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/a8a14/berlin-meetup.jpg 885w,
https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/fbd2c/berlin-meetup.jpg 1180w,
https://bricolage.io/static/0437f7e6c171f69089530459c0d8d109/49ee4/berlin-meetup.jpg 3630w" sizes="(max-width: 590px) 100vw, 590px" loading="lazy" decoding="async">
  </a>
    </span></p>
<p>In the months since, I’ve continued to tinker with these technologies and collected some point-in-time notes on significant developments and what might happen in the years to come.</p>
<h3 id="table-of-contents"><a href="#table-of-contents" aria-label="table of contents permalink" class="anchor before"></a>Table of Contents</h3>

<h2 id="whats-happening"><a href="#whats-happening" aria-label="whats happening permalink" class="anchor before"></a>What’s Happening?</h2>
<p>The web feels ready for a major upgrade. We had tightly coupled web frameworks in the Rails/Django years and lost them with the shift to API-powered SPAs. The developing database-grade sync technology will tightly recouple our application stacks allowing for a new era of framework innovation.</p>
<p>I see “local-first” as shifting reads and writes to an embedded database in each client via“sync engines” that facilitate data exchange between clients and servers. Applications like Figma and Linear pioneered this approach, but it’s becoming increasingly easy to do. The benefits are multiple:</p>
<ul>
<li>Simplified state management for developers</li>
<li>Built-in support for real-time sync, offline usage, and multiplayer collaborative features</li>
<li>Faster (60 FPS) CRUD</li>
<li>More robust applications for end-users</li>
</ul>
<p>(Some good reading material: this local-first case study on <a href="https://riffle.systems/essays/prelude/">building reactive, data-centric apps</a> and <a href="https://www.youtube.com/watch?v=jxuXGeMJsBU&amp;t=1s">Johannes’ talk in Berlin</a>; I will include some more links later.)</p>
<p>Like the shift to componentized JavaScript UI over the last decade, I believe local-first will be the next large paradigm shift for rich client apps and work its way through the application world over the next decade</p>
<h2 id="why-is-local-first-happening-now"><a href="#why-is-local-first-happening-now" aria-label="why is local first happening now permalink" class="anchor before"></a>Why is Local-First Happening Now?</h2>
<p>I’ve read about and played with local-first type ideas for a decade or so but only now does it seem to be gaining steam with many young startups rushing to productize its ideas.</p>
<p>Figma, Superhuman, and Linear are good examples of pioneering startups in the local-first paradigm and all rely on local-first ideas and bespoke sync engines to support their speed and multiplayer UX.</p>
<p>Many builders now see local-first as a key way to differentiate their applications.</p>
<p>Why is this happening now?</p>
<p>My general model for change in technology is that a given community of practice (like application developers) can only adopt one large paradigm shift at a time. While local-first ideas have been floating around for decades, practitioners have so far been focused on more fundamental changes.</p>
<p>What we’ve seen over the last decade is that application speed and collaboration features are powerful vectors to shake up an incumbent industry. Figma used local-first to displace Sketch and InVision; Linear is using local-first to displace Jira.</p>
<p>We’ve shifted from Rails-type server-rendered apps, to single-page-apps powered by APIs. A core lesson from this transition is that while standard REST and GraphQL APIs are very easy to get started with for solving client/server sync, they require significant effort and skill to scale and refine and they struggle with use-cases like multiplayer and offline support.</p>
<p>Sync engines are a robust database-grade syncing technology to ensure that data is consistent and up-to-date. It’s an ecosystem-wide refactor that many talented groups are exploring to attempt to simplify the application stack.</p>
<p>Assuming they succeed, they’ll provide a solid substrate for new types of framework that can rely on local data and rock solid sync.</p>
<h2 id="will-most-rich-client-apps-use-local-first"><a href="#will-most-rich-client-apps-use-local-first" aria-label="will most rich client apps use local first permalink" class="anchor before"></a>Will Most Rich Client Apps Use Local-First?</h2>
<p>I’ve chatted with a number of developers who — frustrated at maintaining the homegrown bespoke sync systems they wrote — are replacing it with new local-first tooling. The tooling feels a lot better and having standard primitives make it easier to build great experiences.</p>
<p>Local-first is developing into an ecosystem similar to authentication services but for handling data and features that need to be real-time, collaborative, or offline.</p>
<p>The key question for us technologists is whether local-first will remain a niche technology or if it’ll gradually replace the current API-based approach.</p>
<p>It’s still early but I’m confident that at a minimum, we’ll see multiple breakout startups along with a few healthy open-source ecosystems around the different approaches. And if local-first becomes the new default paradigm for handling data,, it will be much larger and reshape many parts of the cloud ecosystem.</p>
<p>But: there are many issues to solve first! Let’s look in detail at one of the first issues that people encounter: handling CRUD operations.</p>
<h3 id="crud-with-crdt-based-sync-engines"><a href="#crud-with-crdt-based-sync-engines" aria-label="crud with crdt based sync engines permalink" class="anchor before"></a>CRUD with CRDT-based Sync Engines</h3>
<p>To fully replace client-server APIs, sync engines need robust support for fine-grained access control and complex write validation.</p>
<p>The most basic use case for an API is <em>state transfer</em> from the server to the client. The client wants to show information about an object so reads the necessary data through the API.</p>
<p>Local-first tools all handle this perfectly. They ensure the latest object data is synced correctly to the client db for querying from the UI.</p>
<p>But while reads are generally easy, support for complex writes is still immature in local-first tooling. Clients tend to have unrestricted write access and updates are immediately synced to other clients. While this is generally fine for text collaboration or multiplayer drawing, this wouldn’t work for a typical ecommerce or SaaS application.</p>
<p>Local-first tools need somewhere to put arbitrary business logic written in code, because real-world systems start out elegant and simple and, over time, accumulate lots of messy, important chunks of logic.</p>
<p>A data property might normally be limited to 20 but Acme Corp paid $50k to get a limit of 100. Or write access needs to be restricted for sensitive information like account balances. Or writes need validation by third party APIs, like a calendar booking system that needs to ask an external calendar API if a time slot is open.</p>
<p>Normally, these inevitable chunks of code and logic live on the server behind an API. And we still need somewhere to put them in a local-first world.</p>
<p>Put differently: local-first CRDT-based sync engines can drive consistency within a system but real-world systems also need an authoritative server which can enforce consistency within external constraints and systems.</p>
<p>Or as Aaron Boodman, a <a href="https://replicache.dev/">Replicache</a> co-founder, put it: “CRDTs converge, but to where?”</p>
<h3 id="using-a-distributed-state-machine-to-handle-complex-writes"><a href="#using-a-distributed-state-machine-to-handle-complex-writes" aria-label="using a distributed state machine to handle complex writes permalink" class="anchor before"></a>Using a Distributed State Machine to Handle Complex Writes</h3>
<p>I asked Anselm Eickhoff and James Arthur (founders of <a href="https://jazz.tools/">Jazz</a> and <a href="https://electric-sql.com/">ElectricSQL</a> respectively, both CRDT-based tools) about how they suggest handling writes that need an authoritative server.</p>
<p>They both suggested emulating API request/response patterns through a distributed state machine running on a replicated object.</p>
<p>So let’s say a client wants to update the user name. It creates the following object:</p>
<div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
  <span class="token property">"machine"</span><span class="token operator">:</span> <span class="token string">"updateUserName"</span><span class="token punctuation">,</span>
  <span class="token property">"state"</span><span class="token operator">:</span> <span class="token string">"requestedUpdate"</span><span class="token punctuation">,</span>
  <span class="token property">"request"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
    <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Fred"</span><span class="token punctuation">,</span>
    <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">123</span><span class="token punctuation">,</span>
    <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1694122496</span>
  <span class="token punctuation">}</span><span class="token punctuation">,</span>
  <span class="token property">"response"</span><span class="token operator">:</span> <span class="token punctuation">{</span><span class="token punctuation">}</span>
<span class="token punctuation">}</span>


<span class="token punctuation">{</span>
  <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Bob"</span><span class="token punctuation">,</span>
  <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">123</span>
<span class="token punctuation">}</span></code></pre></div>
<p>The server listens for writes and upon receiving this one, validates the request and updates the state machine object along with the name on the user object (which again are synced back to clients as the “server” is just another client):</p>
<div class="gatsby-highlight" data-language="json"><pre class="language-json"><code class="language-json"><span class="token punctuation">{</span>
  <span class="token property">"machine"</span><span class="token operator">:</span> <span class="token string">"updateUserName"</span><span class="token punctuation">,</span>
  <span class="token property">"state"</span><span class="token operator">:</span> <span class="token string">"finished"</span><span class="token punctuation">,</span>
  <span class="token property">"request"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
    <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Fred"</span><span class="token punctuation">,</span>
    <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">123</span><span class="token punctuation">,</span>
    <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1694122496</span>
  <span class="token punctuation">}</span>
  <span class="token property">"response"</span><span class="token operator">:</span> <span class="token punctuation">{</span>
    <span class="token property">"error"</span><span class="token operator">:</span> <span class="token boolean">false</span><span class="token punctuation">,</span>
    <span class="token property">"timestamp"</span><span class="token operator">:</span> <span class="token number">1694122996</span>
  <span class="token punctuation">}</span>
<span class="token punctuation">}</span>


<span class="token punctuation">{</span>
  <span class="token property">"name"</span><span class="token operator">:</span> <span class="token string">"Fred"</span><span class="token punctuation">,</span>
  <span class="token property">"id"</span><span class="token operator">:</span> <span class="token number">123</span>
<span class="token punctuation">}</span></code></pre></div>
<p>This has the interesting and useful implication that in-flight requests are synced to other clients and it’s trivial for the server to emit progress updates. E.g. an app that supports a team of users uploading images for some nifty AI enhancements. The client can emit progress updates on the upload and the server as it works through enhancements.</p>
<p>In other words, requests/responses have the same multiplayer, offline, real-time sync properties as the rest of the app.</p>
<p>This pattern can be wrapped up in a standard mutation function e.g. <code class="language-text">const {res, error} = await updateName({ name: "fred" })</code></p>
<p>Jazz also lets you restrict writes to certain object fields to a specific role. Sensitive information would be marked read-only for clients who would need to request the server to update them.</p>
<h3 id="local-first-dx-is-still-being-explored"><a href="#local-first-dx-is-still-being-explored" aria-label="local first dx is still being explored permalink" class="anchor before"></a>Local-First DX is Still Being Explored</h3>
<p>Beyond the question of <em>can</em> you build any application with local-first tools, there’s still the question of whether devs will <em>want</em> to.</p>
<ul>
<li>Do advantages make it worth learning a new stack and migrating applications?</li>
<li>There’s a lot of missing components still — what will a full production DX for a local-first toolchain look like?</li>
<li>How complex will it feel to build a simple app?</li>
<li>Is there enough demand to fund all the new libraries and products that’ll need to be built?</li>
</ul>
<p>The success of Figma, Superhuman, and Linear suggest these issues will get solved in time.</p>
<h2 id="what-approaches-are-people-exploring-now"><a href="#what-approaches-are-people-exploring-now" aria-label="what approaches are people exploring now permalink" class="anchor before"></a>What Approaches are People Exploring Now?</h2>
<p>I’m grouping approaches I see into three broad categories.</p>
<h3 id="1-replicated-data-structures"><a href="#1-replicated-data-structures" aria-label="1 replicated data structures permalink" class="anchor before"></a>1. Replicated Data Structures</h3>
<p>These projects provide support for replicated data structures. They are convenient building blocks for any sort of real-time or multiplayer project. They typically give you APIs similar to native Javascript maps and arrays but which guarantee state updates are replicated to other clients and to the server.</p>
<p>It feels like magic when you can build a simple application and and see changes instantly replicate between devices with no additional work.</p>
<p>Most replicated data structures rely on CRDT algorithms to merge concurrent and offline edits from multiple clients.</p>
<p>There’s a number of open source and hosted projects offering replicated data structures, including the granddaddy in this space, Firebase, plus many newer ones.</p>
<p>These services are great for making parts of an app real-time / multiplayer. E.g. a drawing surface, a chat room, a notification system, presence, etc. They’re very simple to get started with and the shared data structures approach offers a much better DX than manually passing events through with websockets or push messaging services.</p>
<p>Open source projects:</p>

<p>Hosted services:</p>

<h3 id="2-replicated-database-tables"><a href="#2-replicated-database-tables" aria-label="2 replicated database tables permalink" class="anchor before"></a>2. Replicated Database Tables</h3>
<p>An approach several projects are taking is to sync from Postgres to a client db (generally SQLite). You pick tables (or materialized views) to sync to the client and then they get loaded along with real-time updates as writes land in Postgres.</p>
<p>SQLite in the browser is one big advantage of this approach as it gives you the rich, familiar querying power of SQL in the client.</p>
<p>Given Postgres’ widespread usage and central position in most application architectures, this is a great way to start with local-first. Instead of syncing data in and out of replicated data structures, you can read and write directly to Postgres as normal, confident that clients will be in sync.</p>
<p>I’ve built a number of job queues and notification systems over the years and they’ve all struggled with their version of the Byzantine Generals problem. Clients would miss an update (usually due to being offline), and then users would complain about zombie jobs that never finished. In contrast, replicated database tables mean the background process can simply write updates to Postgres, confident all connected clients will get the update.</p>
<h4 id="postgres-to-sqlite"><a href="#postgres-to-sqlite" aria-label="postgres to sqlite permalink" class="anchor before"></a>Postgres to SQLite:</h4>

<p>ElectricSQL and Powersync support syncing client writes back to Postgres and other clients.</p>

<ul>
<li>
<p>SQLite to SQLite</p>

</li>
<li>
<p>MongoDB to Client DB</p>

</li>
</ul>
<h3 id="3-replication-as-a-protocol"><a href="#3-replication-as-a-protocol" aria-label="3 replication as a protocol permalink" class="anchor before"></a>3. Replication as a Protocol</h3>
<p>The startup <a href="https://replicache.dev/">Replicache</a> has a unique “replication as a protocol” approach. Replicache is a client JS library along with a replication protocol — which lets you integrate arbitrary backends, provided you follow the spec. It’s more upfront work, as the sync engine is “some assembly required”, but as Replicache is mostly your own code, it gives the most flexibility and power of any local-first tool I’ve seen to date. The startup behind Replicache is also building Reflect, a hosted backend for Replicache.</p>
<h2 id="soshould-you-go-local-first"><a href="#soshould-you-go-local-first" aria-label="soshould you go local first permalink" class="anchor before"></a>So…Should You Go Local-First?</h2>
<p>I think it depends on your use case and risk tolerance.</p>
<p>For the right people and teams, it’s an exciting time to jump in. Realtime, multiplayer, and offline features will significantly improve much of our day-to-day software.</p>
<p>For almost any <strong>real-time use case</strong>, I’d choose <em>replicated data structures</em> over raw web sockets as they give you a much simpler DX and robust guarantees that clients will get updates.</p>
<p>For <strong>multiplayer and offline</strong>, again you’ll almost certainly want to pick an open source <em>replicated data structure</em> or hosted service. There’s a lot of difficult problems that they help solve.</p>
<p>The <em>Replicated Database</em> approach also works for the real-time, multiplayer, offline use cases. It should be especially useful for data-heavy applications and ones with many background processing writing into Postgres.</p>
<p>But in general, I’d still be wary of using local-first outside real-time / multiplayer / offline use cases. Local-first is definitely still bleeding-edge. You will hit unexpected problems. A good community has rapidly developed, but there’ll still be some stretches on the road where you’ll have to solve novel problems.</p>
<p>So: if you need local-first, see if it makes sense to isolate the local-first parts and architect the rest of the app (for now) in a more conventional fashion.</p>
<p>I’ve found that most of the major tools have plenty of examples and demos to play with and active Discord channels. There’s also a general local-first community over at <a href="https://localfirstweb.dev/">https://localfirstweb.dev/</a> and they hold regular meetups.</p>
<p>Following along, I’m getting a lot of the same vibes that I got in the then-nascent React community circa 2014 or 2015.</p>
<p>There are also a lot of interesting implications of local-first development in the realm of privacy, decentralization, data control and so on, but I’ll leave it to others more well-versed in these topics to flesh them out.</p>
<p>And some more links below. Happy building!</p>

<p>Discuss this post on <a href="">X.com née Twitter</a>, <a href="https://warpcast.com/kam/0x0c1632">Farcaster</a>, or <a href="https://bsky.app/profile/kam.bsky.social/post/3k77qihvoip2h">Bluesky</a></p>
<p><em>Thanks to Sam Bhagwat, Shannon Soper, Johannes Schickling, Andreas Klinger, Pekka Enberg, Anselm Eickhoff, and James Arthur for reading drafts of this post</em></p>