? The Cost of Frameworks (archive)

Source originale du contenu

Update: Nov 16th 2015 - Added an extra row in the table for React under production conditions. The good news: it’s 3x slower than vanilla, yes, but in actual terms I’d say it’s fast for TodoMVC! The Polymer TodoMVC sample was also updated to version 1.2.2 today, and that, too, is faster.

If you prefer watching to reading, here’s the video of the talk (you can also get the slides, too, if you like):

If you prefer reading to watching, well, keep reading…

The benefits of frameworks

Earlier in the year I wrote about React’s performance characteristics as the tree size it has to manage gets larger (TL;DR the bigger the tree, the more computation work it has to do). The responses I got to that post really varied, from appreciative and constructive to, well, the other end of the spectrum. The responses gave me some additional insight, though, which I think applies to all frameworks (to a greater or lesser extent), and which I’ll now try and summarize:

The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers.

The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers. If I can paraphrase, the sentiment was “if I have an easier time I’ll build something better for my users”. At face value that sounds appealing, but I think it misses a lot of important points.

Developer Ergonomics vs User Needs

Users have needs, too, though. Here are a few of them that come to mind:

So immediately we’re into a potential trade-off situation: on the one hand we have our convenience, our ergonomics, and on the other hand we have our users’ needs. And we have to trade, whether we like it or not, since frameworks aren’t free.

A quick side step

At this point I’m going to park libraries, and leave them out of this discussion. The reason is that, in my view at least, libraries can be swapped out and replaced if they prove problematic. Don’t like the way a library does date formatting? No problem, just switch it for something else. Frameworks, on the other hand, tend to be a lot more difficult to swap, often requiring a rebuild of the app in question. And, by their nature, they’re a lot bigger and more involved.

The cost of code

All code has costs, but I think there are particular costs that are unique to frameworks.

A deprecation warning.
That magical moment where an framework tells you something is deprecated. Apparently it was superseded by Java. Weird.

Aside from the developer costs, there are costs to the user, too:

Bringing data to the table

So with all that in mind, I wanted to start off a conversation about measuring some of those costs, and my hope is that together we can build up more of a picture about the trade-offs we choose to make.

I decided to take a look at the bootstrapping time for frameworks on mobile. The theory being that, given a like-for-like app, one should be able to see how long it takes each framework to boot. I opted to use TodoMVC for the test, because in my view it’s an MVP web app, and from a user’s point of view, each variant is functionally identical.

TodoMVC.
TodoMVC: source of my bootstrapping test.

Bootstrapping / Time-to-interactive

Of the costs listed above, I decided to take a look at the Time, Bandwidth, and CPU usage for booting up some frameworks on a Nexus 5 and iPhone 5S.

Measuring time to interactive.
Time to interactive: measuring how long it takes for an app to get to a state where someone can interact with it.

I decided to measure the time it takes to evaluate and execute each framework’s initial payload of JavaScript, and the processing and setup of the initial data set into a model. I decided to ignore the cost of styles, layout, paint and so on since they shouldn’t vary all that much between the different TodoMVC implementations. I was also not testing transfer time.

Methodology

I used WebPagetest to load the pages on a Nexus 5, and I requested a timeline file for each run, which I then passed to Big Rig for processing. The iPhone results I did myself, and had to tally up manually, because unfortunately Safari’s JavaScript profiling doesn’t seem to support any timeline / trace file export.

Using Big Rig to measure TodoMVC bootup time.
Using Big Rig to quickly assess the TodoMVC bootup time per framework.

This, incidentally, is why I built Big Rig. I wanted to make it easier for all of us to start processing this kind of data quickly.

Repeating the test

If you want to do your own test like this, at least for a device running Chrome, you should do the following:

If you prefer, you can watch me getting a Timeline file from WebPagetest:

The results

Here’s how the various frameworks stacked up in my tests:

Framework Size Bootstrap time N51,3 Bootstrap time iPhone 5S2,3
Polymer v1.1.4 41KB5 409ms 233ms
Polymer v1.2.2 47KB5 155ms 232ms
AngularJS v1.4.3 324KB 518ms 307ms
React v0.13.3 [JSX not transformed] 311KB 1,201ms 1,463ms
React v0.13.3 [JSX transformed via Babel]4 162KB 509ms 282ms
React v0.13.3 [JSX transformed; prod build]4, 6 160KB 174ms 118ms
Backbone v1.2.2 [inc. jQuery & Underscore] 139KB 248ms 168ms
Ember v1.10.0-beta.3 580KB 1,992ms 1,440ms
Vanilla 16KB 50ms 33ms
  1. Tests done on a Nexus 5 running Chrome 47.
  2. Tests done on an iPhone 5S running Safari 9.
  3. All bootstrapping time includes handling initial todo list data.
  4. JS Transformer stripped; JSX files transformed via Babel.
  5. Excludes Web Components Polyfill (38KB).
  6. React switched for minified production build.

For me the results are pretty clear: there appears to be a pretty hefty tax to using Frameworks on mobile, especially compared to writing vanilla JavaScript. The fastest is React (under production conditions), which is awesome, but it’s still 3x slower than Vanilla (and I still have concerns over its scaling properties.)

To make things super clear, here are a couple of notes:

Possible Objections

I figure there’s likely to be some objections to the test, which are definitely worth talking through:

I’d argue your use-case is probably more expensive than TodoMVC.

The $64k question

So, inevitably, the question comes: “Should you use a framework?”

I can’t answer that question, because I think it’s entirely your call. There are a million and one reasons why you may feel you need to use one. But, for what it’s worth, here are my thoughts:

Investing in knowledge of the web platform itself is the best long-term bet.

As I look at frameworks, I see the ergonomic benefits (and those are important, I agree!), but I can’t help but feel that, for many developers, investing in knowledge of the web platform itself is the best long-term bet. Frameworks come and go, it just seems to be the ebb and flow of the web, and, as I said above, they do contribute ideas and patterns. But if you ever find that the one you use no longer works for you, or has a bug that remains unfixed, being able to understand the platform that underpins it will help enormously.

The bigger conversation

When I wrote the post earlier in the year about React, I said this:

It seems to me that developer ergonomics should be less important than our users’ needs.

I still believe that. As much as I love the idea of having an easier life, I don’t want to ship something to people that doesn’t run well, and I don’t want them to pay the cost. Today I’m concerned by the costs of booting up frameworks on mobile.

I’m concerned by the costs of booting up frameworks on mobile.

This is only a first step. There are other metrics besides bootstrapping I’ve barely had chance to look into: memory usage, long-term CPU usage, and impact on frame rate to name three. Overall, I think we have more to do to properly assess the impact of code that we ship to our users and the costs we pass on to them.

If we can achieve fast-booting, low-memory, smooth-executing frameworks with good ergonomics we’ll be onto a winner. Until then, for mobile at least, I’ll be sticking to the vanilla web platform.