Update: Nov 16th 2015 - Added an extra row in the table for React under production conditions. The good news: it’s 3x slower than vanilla, yes, but in actual terms I’d say it’s fast for TodoMVC! The Polymer TodoMVC sample was also updated to version 1.2.2 today, and that, too, is faster.
If you prefer watching to reading, here’s the video of the talk (you can also get the slides, too, if you like):
If you prefer reading to watching, well, keep reading…
Earlier in the year I wrote about React’s performance characteristics as the tree size it has to manage gets larger (TL;DR the bigger the tree, the more computation work it has to do). The responses I got to that post really varied, from appreciative and constructive to, well, the other end of the spectrum. The responses gave me some additional insight, though, which I think applies to all frameworks (to a greater or lesser extent), and which I’ll now try and summarize:
The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers.
The key message I heard over and over, sometimes explicitly, and often implicitly, is that ergonomics are the most important factor for many developers. If I can paraphrase, the sentiment was “if I have an easier time I’ll build something better for my users”. At face value that sounds appealing, but I think it misses a lot of important points.
Users have needs, too, though. Here are a few of them that come to mind:
So immediately we’re into a potential trade-off situation: on the one hand we have our convenience, our ergonomics, and on the other hand we have our users’ needs. And we have to trade, whether we like it or not, since frameworks aren’t free.
At this point I’m going to park libraries, and leave them out of this discussion. The reason is that, in my view at least, libraries can be swapped out and replaced if they prove problematic. Don’t like the way a library does date formatting? No problem, just switch it for something else. Frameworks, on the other hand, tend to be a lot more difficult to swap, often requiring a rebuild of the app in question. And, by their nature, they’re a lot bigger and more involved.
All code has costs, but I think there are particular costs that are unique to frameworks.
Aside from the developer costs, there are costs to the user, too:
So with all that in mind, I wanted to start off a conversation about measuring some of those costs, and my hope is that together we can build up more of a picture about the trade-offs we choose to make.
I decided to take a look at the bootstrapping time for frameworks on mobile. The theory being that, given a like-for-like app, one should be able to see how long it takes each framework to boot. I opted to use TodoMVC for the test, because in my view it’s an MVP web app, and from a user’s point of view, each variant is functionally identical.
Of the costs listed above, I decided to take a look at the Time, Bandwidth, and CPU usage for booting up some frameworks on a Nexus 5 and iPhone 5S.
I decided to measure the time it takes to evaluate and execute each framework’s initial payload of JavaScript, and the processing and setup of the initial data set into a model. I decided to ignore the cost of styles, layout, paint and so on since they shouldn’t vary all that much between the different TodoMVC implementations. I was also not testing transfer time.
I used WebPagetest to load the pages on a Nexus 5, and I requested a timeline file for each run, which I then passed to Big Rig for processing. The iPhone results I did myself, and had to tally up manually, because unfortunately Safari’s JavaScript profiling doesn’t seem to support any timeline / trace file export.
This, incidentally, is why I built Big Rig. I wanted to make it easier for all of us to start processing this kind of data quickly.
If you want to do your own test like this, at least for a device running Chrome, you should do the following:
npm install -g bigrig.bigrig --file /path/to/timeline.json --pretty-print.If you prefer, you can watch me getting a Timeline file from WebPagetest:
Here’s how the various frameworks stacked up in my tests:
| Framework | Size | Bootstrap time N51,3 | Bootstrap time iPhone 5S2,3 |
|---|---|---|---|
| Polymer v1.1.4 | 41KB5 | 409ms | 233ms |
| Polymer v1.2.2 | 47KB5 | 155ms | 232ms |
| AngularJS v1.4.3 | 324KB | 518ms | 307ms |
| React v0.13.3 [JSX not transformed] | 311KB | 1,201ms | 1,463ms |
| React v0.13.3 [JSX transformed via Babel]4 | 162KB | 509ms | 282ms |
| React v0.13.3 [JSX transformed; prod build]4, 6 | 160KB | 174ms | 118ms |
| Backbone v1.2.2 [inc. jQuery & Underscore] | 139KB | 248ms | 168ms |
| Ember v1.10.0-beta.3 | 580KB | 1,992ms | 1,440ms |
| Vanilla | 16KB | 50ms | 33ms |
For me the results are pretty clear: there appears to be a pretty hefty tax to using Frameworks on mobile, especially compared to writing vanilla JavaScript. The fastest is React (under production conditions), which is awesome, but it’s still 3x slower than Vanilla (and I still have concerns over its scaling properties.)
To make things super clear, here are a couple of notes:
I figure there’s likely to be some objections to the test, which are definitely worth talking through:
I’d argue your use-case is probably more expensive than TodoMVC.
So, inevitably, the question comes: “Should you use a framework?”
I can’t answer that question, because I think it’s entirely your call. There are a million and one reasons why you may feel you need to use one. But, for what it’s worth, here are my thoughts:
Investing in knowledge of the web platform itself is the best long-term bet.
As I look at frameworks, I see the ergonomic benefits (and those are important, I agree!), but I can’t help but feel that, for many developers, investing in knowledge of the web platform itself is the best long-term bet. Frameworks come and go, it just seems to be the ebb and flow of the web, and, as I said above, they do contribute ideas and patterns. But if you ever find that the one you use no longer works for you, or has a bug that remains unfixed, being able to understand the platform that underpins it will help enormously.
When I wrote the post earlier in the year about React, I said this:
It seems to me that developer ergonomics should be less important than our users’ needs.
I still believe that. As much as I love the idea of having an easier life, I don’t want to ship something to people that doesn’t run well, and I don’t want them to pay the cost. Today I’m concerned by the costs of booting up frameworks on mobile.
I’m concerned by the costs of booting up frameworks on mobile.
This is only a first step. There are other metrics besides bootstrapping I’ve barely had chance to look into: memory usage, long-term CPU usage, and impact on frame rate to name three. Overall, I think we have more to do to properly assess the impact of code that we ship to our users and the costs we pass on to them.
If we can achieve fast-booting, low-memory, smooth-executing frameworks with good ergonomics we’ll be onto a winner. Until then, for mobile at least, I’ll be sticking to the vanilla web platform.