A place to cache linked articles (think custom and personal wayback machine)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

index.md 18KB

11 months ago
  1. title: Deno is a Browser for Code
  2. url: https://kitsonkelly.com/posts/deno-is-a-browser-for-code/
  3. hash_url: e0ed8bab5c145d37065c607deace5bff
  4. <p>I started contributing to Deno soon after Ry made the prototype visible in
  5. May 2018. The most frequent question that people have is “where is the package
  6. manager?” which often times isn’t even in the form of a question. It is
  7. statements like “I thought Deno took security seriously, and just downloading
  8. resources off the internet is insecure.” or “How can I possibly manage my
  9. dependencies?”</p>
  10. <p>In my opinion, we need to shift our mental model. Lots of folks take the
  11. ubiquity of package managers and centralized code registries as a requirement
  12. to have a package manager and a centralized code registries. Because they exist
  13. doesn’t mean they are required. They came into existence because they solved
  14. problems in a particular way, and we have just accepted them as the only way to
  15. solve that problem. I would argue that isn’t true.</p>
  16. <h2 id="browsers">Browsers</h2>
  17. <p>In order to publish a website, we don’t login to a central Google server, and
  18. upload our website to the registry. Then if someone wants to view our website,
  19. they use a command line tool, which adds an entry to our <code>browser.json</code> file on
  20. our local machine and goes and fetches the whole website, plus any other
  21. websites that the one website links to to our local <code>websites</code> directory before
  22. we then fire up our browser to actually look at the website. That would be
  23. insane, right? So why accept that model for running code?</p>
  24. <p>The Deno CLI works like a browser, but for code. You import a URL in the code
  25. and Deno will go and fetch that code and cache it locally, just like a browser.
  26. Also, like a browser, your code runs in a sandbox, which has zero trust of the
  27. code you are running, irrespective of the source. You, the person invoking the
  28. code, get to tell that code what it can and can’t do, externally. Also, like a
  29. browser, code can ask you permission to do things, which you can choose to grant
  30. or deny.</p>
  31. <p>The HTTP protocol provides everything that is needed to provide information
  32. about the code, and Deno tries to fully leverage that protocol, without having
  33. to create a new protocol.</p>
  34. <h2 id="discovering-code">Discovering code</h2>
  35. <p>The first thing to think about is that, like a browser, the Deno CLI doesn’t
  36. want to have any opinions about what code you run. It lays out the rules of how
  37. code is fetched, and how it sandboxes itself from the machine it runs on. In my
  38. opinion, that is as much of an opinion a runtime should have.</p>
  39. <p>In the Node.js/npm ecosystem, we have conflated the management of code on our
  40. local machine, with a centralized registry of code to help facilitate discovery.
  41. In my opinion, both have really bad flaws.</p>
  42. <p>Back in the early days of the internet, we experimented with npm type of
  43. discoverability. You would go add your website to Yahoo! under the right
  44. categorization and people would come along, maybe use the search function, but
  45. it was all structured based on the opinions of those providing the content, not
  46. really based on optimizing for the needs of the consumer. Eventually along came
  47. Google. Why did Google win? Because it was useful. It indexed websites in a
  48. way that matched simple expressions of need (search terms) with the most
  49. relevant web pages that met that need, looking at multiple factors, including
  50. meta data provided the content provider as one factor in the mix.</p>
  51. <p>While we don’t have that model quite yet for code for Deno, it is a model that
  52. works. In addition, we use Google because it solves problems for us, instead of
  53. being told “you must use Google”, as well as there are also other viable
  54. alternatives to Google.</p>
  55. <p>I got into a bit of a debate with Laurie Voss on twitter, someone who knows a
  56. fair deal about the npm ecosystem I would say. He argued that Deno needed a
  57. package manager, and this blog post is a longer winded version of the thoughts
  58. I wanted to express, but Laurie raised a very valid point.</p>
  59. <p>GitHub has become the home for open source code, because it was useful and
  60. solved problems, and built on top of the <em>de facto</em> source code versioning tool,
  61. git. From the Deno CLI perspective, there should be no technical restrictions
  62. to where you source code from, it is up to the wider eco-system to create and
  63. evolve ways to make code for Deno discoverable, probably in innovative ways that
  64. could never have been conceived by those of us creating the CLI.</p>
  65. <h2 id="repeatable-builds">Repeatable builds</h2>
  66. <p>In the npm eco-system, this became a problem. Because of the heavy reliance on
  67. semantic versioning, and the complex dependency graphs that tend to come from
  68. the Node.js/npm eco-system, having a repeatable build became a real problem.
  69. Yarn introduced the concept of lock files, of which npm followed suit.</p>
  70. <p>My personal feeling is it was a bit of the tail wagging the dog, in that the
  71. behaviours of developers in the eco-system created a problem that then needed
  72. an imperfect solution to fix it. Any of us that have lived with the eco-system
  73. for a long time know that the fix to a lot of issues is
  74. <code>rm -rf node_modules package-lock.json &amp;&amp; npm install</code>.</p>
  75. <p><img src="https://memegenerator.net/img/instances/75583685/have-you-tried-rm-rf-node-modules-npm-install.jpg" alt=""/></p>
  76. <p>That being said, Deno has two solutions for that. First, is that Deno caches
  77. modules. That cache can be checked into your source control, and the
  78. <code>--cached-only</code> flag will ensure that there is not attempts to retrieve remote
  79. modules. The <code>DENO_DIR</code> environment variable can be used to specify where the
  80. cache is located to provide further flexibility.</p>
  81. <p>Second, Deno supports lock files. <code>--lock lock.json --lock-write</code> would write
  82. out a lock file with hashes of all the dependencies for a given workload. This
  83. would be used to validate future runs when the <code>--lock lock.json</code> is used.</p>
  84. <p>There are also a couple other commands that make managing repeatable builds.
  85. <code>deno cache</code> would resolve all the dependencies for a supplied module and
  86. populate the Deno cache. <code>deno bundle</code> can be used to generate a single file
  87. “build” of a workload which all the dependencies are resolved and included in
  88. that file, so only that single file is needed for future <code>deno run</code> commands.</p>
  89. <h2 id="trusting-code">Trusting code</h2>
  90. <p>This is another area where I think we have a skewed mental model. For whatever
  91. reason, we put trust in code that is in a centralized registry. We don’t even
  92. think about it. Not only that, we trust that that code has fully vetted all of
  93. its dependencies and that those are to be trusted to. We do a quick search and
  94. type in <code>npm install some-random-package</code> and think “This is Fine!” I argue the
  95. rich npm package eco-system has lulled is into a sense of complacency.</p>
  96. <p>To compensate for this laxness and complacency, we implement security monitoring
  97. software in our tool chains, to analyse our dependencies and the thousands upon
  98. thousand lines of code to let us know that maybe some of the code is
  99. exploitable. Corporations setup private registries to host packages that might
  100. be vetted slightly more than the single public registry.</p>
  101. <p>It feels like there is an elephant in the room here. The best strategy is we
  102. shouldn’t trust any code. Once we have that established, then opening it back
  103. up becomes a little be easier. But we are lying to ourselves if we think a
  104. package manager and a centralised registry solve this problem, or even
  105. substantially help with this problem. In fact, I argue they make use let our
  106. guards down. “Well it is on npm, if it were bad for me, surely someone would
  107. take it down.”</p>
  108. <p>Deno in this aspect isn’t quite as done as I think it should be, but it is
  109. starting from a good position. It has zero trust at startup, and provides
  110. fairly fine grained permissions. One of the things I personally dislike is that
  111. there is the <code>-A</code> flag, which is basically saying “oh yeah allow everything”
  112. which is such an easy thing for a frustrated developer to do instead of figuring
  113. out what they really need.</p>
  114. <p>It is also hard to break down those permissions, to say “this code can do this,
  115. but this other code over here can’t” or when code prompts to escalate privileges
  116. where is that code coming from. Hopefully we can figure out an easy to use
  117. mechanism coupled with something that would be effective and performant at
  118. runtime to try to solve those challenges.</p>
  119. <p>A recent change though, which is a good one, in my opinion, is that Deno no
  120. longer allows you to downgrade your imports. If something is imported from
  121. <code>https://</code> then it can only import from other <code>https://</code> locations. This
  122. follows the browser model of not being able to downgrade transport. I still
  123. think longer term it would be good to kill off any remote imports that aren’t
  124. over <code>https://</code>, much like Service Workers require HTTPS, so we will see what
  125. the future holds.</p>
  126. <h2 id="dependency-management">Dependency management</h2>
  127. <p>I think we need to talk frankly about dependencies in the npm ecosystem. To be
  128. honest, it is broken. An ecosystem that enables
  129. <a href="https://github.com/juliangruber/isarray/blob/master/index.js" target="_blank">5 lines of code</a>
  130. to be downloaded and installed
  131. <a href="https://www.npmjs.com/package/isarray" target="_blank"><em>30 million</em> times a week</a> for code that
  132. has been in every browser for the last 9 years and never was needed in Node.js
  133. is a broken ecosystem. This one example, the actual code is 132 bytes, but the
  134. package size is 3.4kb. The runnable code is 3.8% of the package size. “This is
  135. Fine!”</p>
  136. <p>My opinion is that there are several factors involved in this. A big part of it
  137. is that we have the model inverted, which I talked about Deno being a browser
  138. for code. The problem is that this backwards model has infected how we create
  139. websites. While we don’t have a central registry, when we build a website,
  140. we download all the code we depend up and bake it into something that we load
  141. up on a server, and then each user downloads a bunch of code to their local
  142. machine. Some evidence is that only around 10% of that code that is downloaded
  143. is unique to that site or web application, the rest is all that code we are
  144. downloading to our development workstation and bundling up. This model being
  145. broken are some of the problems solutions like
  146. <a href="https://www.snowpack.dev/" target="_blank">Snowpack</a> are trying to solve.</p>
  147. <p>Another significant problem is that our dependencies are not coupled with our
  148. code. We put dependencies in our <code>package.json</code> but if our code actually uses
  149. that code or not is totally decoupled. While our code expresses what we are
  150. using out of that other code, it is very loosely coupled to the version of that
  151. code. That is contained in the <code>package.json</code>, though it has the biggest impact
  152. on the code we write, because it is the code that is actually consuming the
  153. dependent code.</p>
  154. <p>This leads us to the Deno model, which I like to call <em>Deps-in-JS</em>, since all
  155. the cool kids are doing <em>*-in-JS</em> things. Explicitly stating our external
  156. dependencies as URLs means that the code depends upon the other code is concise
  157. and clear, and our code and dependencies are tightly coupled together. If you
  158. want to see that dependency graph, you simply need to use <code>deno info</code> with a
  159. local or remote module:</p>
  160. <pre><code class="language-shell">$ deno info https://deno.land/x/oak/examples/server.ts
  161. local: $deno/deps/https/deno.land/d355242ae8430f3116c34165bdae5c156dca21aeef521e45acb51fcd21c9f724
  162. type: TypeScript
  163. compiled: $deno/gen/https/deno.land/x/oak/examples/server.ts.js
  164. map: $deno/gen/https/deno.land/x/oak/examples/server.ts.js.map
  165. deps:
  166. https://deno.land/x/oak/examples/server.ts
  167. ├── https://deno.land/std@0.53.0/fmt/colors.ts
  168. └─┬ https://deno.land/x/oak/mod.ts
  169. ├─┬ https://deno.land/x/oak/application.ts
  170. │ ├─┬ https://deno.land/x/oak/context.ts
  171. │ │ ├── https://deno.land/x/oak/cookies.ts
  172. │ │ ├─┬ https://deno.land/x/oak/httpError.ts
  173. │ │ │ └─┬ https://deno.land/x/oak/deps.ts
  174. │ │ │ ├── https://deno.land/std@0.53.0/hash/sha256.ts
  175. │ │ │ ├─┬ https://deno.land/std@0.53.0/http/server.ts
  176. │ │ │ │ ├── https://deno.land/std@0.53.0/encoding/utf8.ts
  177. │ │ │ │ ├─┬ https://deno.land/std@0.53.0/io/bufio.ts
  178. │ │ │ │ │ ├─┬ https://deno.land/std@0.53.0/io/util.ts
  179. --snip--
  180. </code></pre>
  181. <p>Deno has no strong opinions around “versions” of code. A URL is a URL is a URL.
  182. While Deno requires an appropriate media type in order to understand how to
  183. treat code, all the “opinions” about what code to serve up is left up to the
  184. web server. A server can implement semantic versioning to its hearts content,
  185. or do any sort of “magical” mapping of URLs to resources it wants. Deno doesn’t
  186. care. For example <code>https://deno.land/x/</code> is effectively nothing but a URL
  187. redirect server, where it rewrites URLs to include a git commit-ish reference
  188. in the redirected URL. So <code>https://deno.land/x/oak@v4.0.0/mod.ts</code> becomes
  189. <code>https://raw.githubusercontent.com/oakserver/oak/v4.0.0/mod.ts</code>, which GitHub
  190. serves up a nice versioned module.</p>
  191. <p>Of course spreading “versioned” remote URLs throughout your codebase doesn’t
  192. make a lot of sense, so don’t do that. The great thing about the dependencies
  193. just being code though is that you can structure them any way you want to. A
  194. common convention is to use a <code>deps.ts</code> which re-exports all the dependencies
  195. you might want. Take a look at the one for
  196. <a href="https://deno.land/x/oak@v4.0.0/deps.ts" target="_blank">oak server</a>:</p>
  197. <pre><code class="language-ts">// Copyright 2018-2020 the oak authors. All rights reserved. MIT license.
  198. // This file contains the external dependencies that oak depends upon
  199. // `std` dependencies
  200. export { HmacSha256 } from "https://deno.land/std@0.51.0/hash/sha256.ts";
  201. export {
  202. Response,
  203. serve,
  204. Server,
  205. ServerRequest,
  206. serveTLS,
  207. } from "https://deno.land/std@0.51.0/http/server.ts";
  208. export {
  209. Status,
  211. } from "https://deno.land/std@0.51.0/http/http_status.ts";
  212. export {
  213. Cookies,
  214. Cookie,
  215. setCookie,
  216. getCookies,
  217. delCookie,
  218. } from "https://deno.land/std@0.51.0/http/cookie.ts";
  219. export {
  220. basename,
  221. extname,
  222. join,
  223. isAbsolute,
  224. normalize,
  225. parse,
  226. resolve,
  227. sep,
  228. } from "https://deno.land/std@0.51.0/path/mod.ts";
  229. export { assert } from "https://deno.land/std@0.51.0/testing/asserts.ts";
  230. // 3rd party dependencies
  231. export {
  232. contentType,
  233. lookup,
  234. } from "https://deno.land/x/media_types@v2.3.1/mod.ts";
  235. </code></pre>
  236. <p>I created oak server and maintained for 18 months through about 40 releases of
  237. Deno and the Deno <code>std</code> library, including moving of <code>media_types</code> from internal
  238. to oak, out to the <code>std</code> library, to only have it be “ejected” from the <code>std</code>
  239. library to be its own thing. Not once did I think to myself “hey, I need a
  240. package manager to manage this for me”.</p>
  241. <p>One of the benefits of TypeScript is that you can get comprehensive validation
  242. of compatibility of your code with other code. If your dependencies are “raw”
  243. TypeScript written for Deno, this is great, but let’s say that you want to take
  244. advantage of pre-processing of the TypeScript to JavaScript, but still have the
  245. ability to consume that remote code safely. Deno supports a couple different
  246. ways to allow that to happen, but the most seamless is the support for the
  247. <code>X-TypeScript-Types</code> header. This header indicates to Deno where a types file
  248. is located which can be used when type checking the JavaScript file that you
  249. are depending upon. <a href="https://pika.dev/cdn" target="_blank">Pika CDN</a> supports this. Any
  250. packages that are available on the CDN that have types associated with them will
  251. serve up that header and Deno will also fetch those types and use that when
  252. type checking the file.</p>
  253. <p>All this being said, there may still be a need to “remap” a remote (or local)
  254. dependency to what is expressed in the code. In this case, the unstable
  255. implementation of <a href="https://github.com/WICG/import-maps" target="_blank">import-maps</a> can be
  256. used. It is a proposal specification that is part of the W3C incubator where
  257. browser standards come out of. It allows a map to be provided which will map
  258. a particular dependency in code to another resource, be it a local file or a
  259. remote module.</p>
  260. <p>We had it implemented in Deno for an extended period of time, as we had really
  261. hoped that it would become adopted widely. Sadly, it was only an
  262. <a href="https://chromestatus.com/feature/5315286962012160" target="_blank">origin trial in Chrome</a> and
  263. hasn’t gotten wider adoption yet. This led us to putting it behind the
  264. <code>--unstable</code> flag for Deno 1.0. My personal opinion is that it is still a big
  265. risk of being a dead end, and should be avoided.</p>
  266. <h2 id="but-but-but">But, but, but…</h2>
  267. <p>I suspect a lot of people are still coming with a list of objections to the
  268. model that Deno has. I think the strategy Deno has tried to take, which I am
  269. very aligned to, is to deal with real problems when they arise. A lot of the
  270. objections I hear are from people who are new to Deno, who haven’t worked with
  271. it, who haven’t tried to understand that there might be a different way.</p>
  272. <p>All that being said, if we collectively run into a problem and there is a
  273. compelling need to change something in the Deno CLI, I am confident that it will
  274. happen, but a lot of problems simply don’t exist, or there are other ways to
  275. solve them that don’t require your runtime to have strong opinions or be coupled
  276. to an external programme to manage your code.</p>
  277. <p>So my challenge to you is, flirt a bit with not having a package manager or
  278. a centralised package repository and see how it goes. You might never go back!</p>