larlet-fr-david-cache/cache/f8debee9c305a9fb2da16ac87f87a371/index.md

title: Bringing the DAT protocol to Firefox, part 1
url: https://sammacbeth.eu/blog/2019/03/22/dat-for-firefox-1.html
hash_url: f8debee9c305a9fb2da16ac87f87a371

<p>The Dat protocol enables peer-to-peer sharing of data and files across the web. Like similar technologies such as IPFS and Bittorrent it allows clients to validate the data received, so one can know that the data is replicated properly, but in contrast Dat also supports modification of the resources at a specific address, with fast updates propagated to peers. Other useful properties include private discovery - allowing data shared privately on the network to remain so.</p>

<p>These features have led to a movement to use it has a new protocol for the web, with the <a href="https://beakerbrowser.com/">Beaker browser</a> pushing innovation around what this new peer-to-peer web could look like. The advantages of using Dat for the web are many-fold:</p>

<ul>
  <li>Offline-first: Every site you load is implicitly kept locally, allowing it to be navigated when offline. Similarly, changes to sites (both local and remote) will propagate when connectivity is available, but functionality will always be the same.</li>
  <li>Transparent and censorship resistant: Sites are always the same for every user - the site owner cannot decide to change site content based on your device or location as is common on the current web. As sites are entirely published in Dat, and there is no server-side code, then all the code running the site can be seen with ‘view source’.</li>
  <li>Self-archiving: Dat versions all mutations of sites, so as long as at least one peer keeps a copy of this data, the history of the site will remain accessible and viewable. This can also keep content online after the original publisher stops serving their content.</li>
  <li>Enables self-publishing: As servers are no longer required, anyone can push a site with DAT - no server or devops required. Publishing to the P2P web requires no payment, no technical expertise, and no platform lock-in.</li>
  <li>Resilient: Apps and sites stay up as long as people are using them, even if the original developers have stopped hosting.</li>
</ul>

<p>The Beaker browser already demonstrates all of these features, but as an electron-based app lacks some of the security features, depth of configuration and extensibility of a fully-fledged browser. For this reason I wanted to explore how we could bring these features to Firefox, and could enable access to the Dat web for the low cost of installing a browser extension. (*Also as I work for a company building a fork of Firefox, I have a vested interest in getting this working in the browser I develop).</p>

<p>This article is split into two parts. The first part describes the dat-fox, a Firefox extension that provides the best-possible Dat support that is possible given the current limitations of the <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions">webextensions APIs</a>. The second describes the process and challenges in creating <a href="https://github.com/cliqz-oss/dat-webext">dat-webext</a> a Firefox extension which uses experimental APIs from the <a href="https://github.com/mozilla/libdweb">libdweb</a> project to build full Dat protocol support into a Webextension, and which is current bundled with the <a href="https://cliqz.com/en/latest">Cliqz Browser nightly</a> build.</p>

<p>There were three main challenges to building Dat support in an extension:</p>

<ol>
  <li>Running Dat in an extension context. Dat is currently only implemented for nodejs (though a Rust implementation is on the way), and uses APIs such as <code class="highlighter-rouge">net</code> and <code class="highlighter-rouge">dgram</code> which have no analogues in the web-stack. This means that we need to find a way to running this implementation in a webextension, or  find alternative ways of communicating with other peers to get the content of DAT sites.</li>
  <li>Adding new protocols to the browser, such that it can understand an address starting with <code class="highlighter-rouge">dat://</code> . This then has to be wired with the Dat implementation to return the correct content for that URL such that it can be rendered in the browser.</li>
  <li>Adding new web APIs for Dat. In Beaker, a new API, <code class="highlighter-rouge">DatArchive</code> was proposed, which allows pages to programmatically read the contents of Dat sites. For sites where the user is the owner, and has write permissions, this API allows writes. This API is innovative as it enables self-mutating sites, and has spawned various ‘Dat Apps’ which behave like many modern web-apps, yet have no server.</li>
</ol>

<h2 id="first-attempt-dat-fox">First Attempt: Dat-fox</h2>

<p>Early last year, inspired by the <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1428446">whitelisting of p2p protocols</a> for use with the Webextensions protocol handlers API, I started <a href="https://github.com/sammacbeth/dat-fox">dat-fox</a> to build <code class="highlighter-rouge">dat://</code>support in a Webextensions compatible extension. Unfortunately, the current APIs are severely limiting, meaning that all three of the above challenges could only be partially solved.</p>

<p>Webextensions allow <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/protocol_handlers">protocol handlers</a> to be specified in their manifest, however these function as simple redirects. To render content under these handlers a HTTP server is still required, either on the web, or running locally. As we also cannot run a HTTP server inside the extension, the APIs necessitate the use of an external process that will serve the content for <code class="highlighter-rouge">dat://</code> URLs.</p>

<p>Dat-fox implemented a <code class="highlighter-rouge">dat://</code> protocol handler which redirected to a local process, launched via the <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Native_messaging">native messaging API</a>. The separate process, written in node, manages syncing with the dat network, and acts as a gateway HTTP server so that the browser can load <code class="highlighter-rouge">dat://</code> pages when redirected.</p>

<p><img src="/assets/images/dat-fox-protocol.png" alt="dat-fox-protocol"/></p>

<p>A further challenge for dat-fox was ensuring the origins of <code class="highlighter-rouge">dat://</code> pages were correct, and that URLs looked correct when browsing. Each Dat archive written as a webpage expects to be on it’s own <em>origin</em>. For example, then page <code class="highlighter-rouge">dat://example.com/test.html</code> should have the origin <code class="highlighter-rouge">example.com</code>. This is important for both the browser’s security model, such that <code class="highlighter-rouge">localStorage</code> is not shared between sites, and also for calculating relative paths to files from a specific document. A naive implementation of the protocol handler might redirect the browser to <code class="highlighter-rouge">http://localhost/example.com/test.html</code>. However, this page would then have the incorrect origin <code class="highlighter-rouge">localhost</code>, and could break links in the rendered page.</p>

<p>We solved this issue by tricking the browser into loading pages like <code class="highlighter-rouge">http://example.com/test.html</code> via the gateway. Using a <a href="https://github.com/sammacbeth/dat-fox/blob/master/addon/pac.js#L23">PAC file</a>, which allows the browser to dynamically send traffic to different proxies, we can take requests to <code class="highlighter-rouge">dat://</code> URLs, after redirecting and tell the browser to use the gateway as a proxy.</p>

<p>Finally, to support the <code class="highlighter-rouge">DatArchive</code> API, this class needed to be added to the <code class="highlighter-rouge">window</code> object on Dat pages. While Webextensions do allow for code to be run in the page context via a content-script, this code is sandboxed. This means any modifications to <code class="highlighter-rouge">window</code> from the content-script are not seen by the page. Instead, we have to use the content-script to firstly inject a script in the page which creates the <code class="highlighter-rouge">DatArchive</code> object. This script then communicates API calls to the content-script via the postmessage API, which in turn relays to the extension background. As Dat operations require the external node process, these must then be further forwarded via native messaging, then the response returned back up the stack. Luckily there are libraries like <a href="https://github.com/chrmod/spanan">spanan</a> which make all this async messaging a bit easier to handle.</p>

<h2 id="conclusion">Conclusion</h2>

<p>While dat-fox does enable browsing the Dat web, multiple limitations of the Webextension API mean that this support is second-class: Users have to install a separate executable to bridge to Dat, and when visiting Dat sites the URL bar still displays <code class="highlighter-rouge">http://</code> as the protocol.</p>

<p>To overcome these limitations we have to extend beyond what a standard Webextension can do, using experimental APIs to bring fully-fledged Dat support. In the next post I’ll describe how <a href="">dat-webext</a> bundles the Dat networking stack inside a Webextension using libdweb networking and protocol handler.</p>