The results given by this benchmark are bothering me because they do not fit with what I've seen in production.
For example, the `for-of-array` is transpiled by babel to something that has 2 nested try/catch blocks, and unfortunately, V8 has a nasty deoptimisation when dealing with these, even if no exception is ever thrown.
This deopt led to a several orders of magnitude slowdown compare to a simple `for` loop in Google Chrome, forcing use to abandon the `for of` construction.
I suspect that the benchmark shown on this page does not expose this kind of behavior because it doesn't iterate enough for the JIT to kick in. If it's the case, it means that these values reflects only the behavior of the cold, interpreted code, and not the hot one. (which is quite sad for a performance benchmark, because the performance matter only for the former …)
Chrome doesn't have an interpreter, it has baseline and optimizing compiler. Also, I think try/catch preventing optimization was fixed.
Still your point could stand, or additionally there could be many optimizations that interfere with such simple micro benchmarks, turning all or parts of them into no-ops. If this is happening in ES5 and not ES6, the results could be drastic. I suspect this is what's happening with some of the larger differences in native ES6.
Agreed, and issues like this make me hugely suspicious of JS performance benchmarks. I'm only familiar with V8, but my experience has been that there are more or less three "tiers" of factors that affect performance:
1. whether the optimizing compiler gets used
2. random stuff that's impossible to know about unless you grab your code's internal representation and decompile it [1]
3. the raw performance of individual statements
The effects of (1) dominate (2), and the effects of (2) dominate (3). So when I see performance micro-benchmarks that purport to measure (3) without apparently paying any attention to (1) or (2), my first inclination is to assume the results are essentially random noise.
[1] E.g. whether statements get wrapped in type checks, whether an integer variable gets handled internally as a SMI (small int) or whether it keeps getting converted to a boxed Number and back. The cost of such things can easily exceed the cost of the statement you're trying to benchmark.
Yes, my understanding is the problem is essentially undefined behavior as exceptions can be caught at any point in the call stack which makes it basically impossible to optimize.
It's not impossible at all. try/catch in the non-throwing case is fast in both Firefox (SpiderMonkey) and IE (Chakra); I'm not sure what the state of things is in Safari (JavaScriptCore). In the throwing case you obviously have to do some work, but as long as that case is rare it's not a problem. Contrast with the V8 situation (which they are fixing), where simply having a try/catch in your function at all will deoptimize the function, even if an exception is never actually thrown.
Oh, and one optimization strategy for try catch is even pretty simple to describe in general terms: you need a cheap way to check whether an exception is thrown, and then after every operation that can throw (e.g. a call into the vm to a function that is allowed to throw) you check whether it did. If not, you just move along. If it did, you jump to an out of line path that does cleanup and callstack unwinding. The devil, as usual, is in the details.
I'm not a frontend developer but... is it still truly necessary to transpile stuff like for-of ?
It's supported by every desktop browser on the market, modulo some bugs (like inability to iterate over HTMLCollection and co. on Chrome)... as long as you only need to use it for arrays, things might work quite fine
Unless you're a startup and don't have to give a damn about what browsers your users use, IE marketshare is just too big to ignore. And depending on your audience, IE can mean anything from IE11 all the way back to IE8.
Theoretically IE is irrelevant now that Microsoft has started publicly killing it off, but you'd be surprised at the number of corporate users that are still on Windows XP (or maybe Vista/7) running IE 8 and 9.
Of course evergreen browsers are less of a problem but amazingly a lot of the same users who are stuck with IE also manage to have extremely outdated versions of Firefox.
The thing about JIT languages is that there's no such thing as overall speed of a particular operation; you can only sample current speed on current implementations.
New features pursue correctness, then as their place in the JITs mature interesting new approaches will change their speed characteristics later. Things like contrasting the speed of various ways of iterating an array are all semantically equivalent, and as the JIT learns the various semantics they should all theoretically approach similar speeds, even if the newer ones are slower now.
> The thing about JIT languages is that there's no such thing as overall speed of a particular operation; you can only sample current speed on current implementations.
How in the world is that specific to JIT? Have you never heard of a Sufficiently Smart Compiler?
The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
I'm not sure this is true anymore even in this case - with multicore processors, cache dependency, and pervasive virtualization, the speed of your program can be significantly affected by what else is running on the box. Remember that the x86 itself has a JIT on the chip, converting x86 machine code to whatever microcode the processor uses. When I was at Google, they provided special dedicated machines for benchmarking, with custom run-times that disabled a lot of the containerization/virtualization features, and there was still a lot of noise in benchmark times.
That's certainly true and I hope I didn't give the impression that I thought this was a binary situation. It's just that code which runs in a JITed environment, particularly one like the browser JavaScript runtimes which aren't even versioned, has an even wider exposure to skew. Everything running on a multiprocess/user operating system is exposed to resource contention but e.g. your JavaScript code also has to worry about things like Chrome disabling optimizations anywhere you use try/catch.
> The main difference is that you the developer can test with a known compiler and know what the performance characteristics are before you release your code: when Microsoft releases VisualStudio 2018, your existing application doesn't change until you rebuild it. In contrast, for code running in a web browser or something like the JVM, your existing code may run faster or slower without you even knowing about the new version.
While the release of a new AOT compiler version might not affect the performance characteristics of your application without an active choice to recompile, the release of updates to any OS components your application interacts with, may, in much the same way that a new web browser or JVM version would. The only way you get immunity from that is to bundle the whole software stack running on the bare metal as your "app", rather than relying on lower-level software that may change independently.
Right – that's why I didn't say “known compiler and standard library”. That said, it's a difficult comparison because OS updates tend to be more conservative and are definitely less frequent than browser updates, AOT apps in lower-level languages like C/C++ are going to manage a higher percentage of their core data structures because they're pinning something like Boost whereas a JavaScript developer just gets whatever version of Object/Array/etc. is available, and there's not really a concept of a pinned release: if you install Windows apps, you'll pull in major versions of things like .NET or the JVM but there's no analogous practice of pinning a particular browser version. The closest we've come was Internet Explorer's compatibility modes which they've deprecated and which never changed the core JavaScript engine's performance – e.g. IE8-running-as-IE7 still applied the IE8 JavaScript engine's validation and optimizations.
It does but the most common JITed language in existence is JavaScript, where the uncertainty is especially pronounced. For example, Python isn't [traditionally] a JITed language but most Python developers know that their code will run on CPython, which is pretty stable and you have a rough idea of which versions it'll run on, whereas something running in a browser can see huge shifts in what is an expensive or cheap operation from user to user over the course of a single day, much less release to release.
Remember when WebKit started JITing regular expressions? That changed the benchmark situation so much that some people called it cheating and not a single line of user code changed. That's not impossible in other environments – I'm sure there's been a case where a shared library update made a big difference – but it's far less common in my experience.
More than just these, even innocuous features such as `let` and `const` over `var` cause a substantial performance decrease in V8 (but not SpiderMonkey).
I do a lot of one-off demos, that I mostly write in ES6 these days. I don't have a demonstration of pure `let` vs `var`, but if you change Babel to JavaScript in this one (and click run), you'll see a fairly substantial performance decrease in Chrome for this demo (Firefox stays the same though): https://jsfiddle.net/oa1sckzu/ . I have others, and the results are largely the same for them, but I'll spare you.
These measures are JS performance are important for VM implementors, for people writing node.js applications to be used at large scale and people doing computationally-complex work in the browser (eg creating physics engines).
It's worth noting that for most of us, most of the time, this kind of JS performance is less important than user-perceptible optimizations, like batching DOM reads and writes and decreasing asset size
I assumed the "es6" row header refers to the engine's native implementation of the feature. In which case it should grey out "destructuring" under Node 4.2.6. But they don't, it's listed as "10x slower". What am I missing?
I would like to see Babel ES6-to-ES5 converted performance here too, because that's how most real-world ES6 code is going to be run --- there are too many non-ES6-compatible browsers out there.
That's exactly the information the "babel" rows provide? Each row is a given "ES6 implementation" and how it performs compared to the baseline "ES5 native" implementation of a feature.
So the first row of each section is the babel-compiled (to ES5) ES6, the second is traceur-compiled, third is typescript-compiled and last is native ES6 runtime. If you look at the test files, most of them only have an es5 and an es6 versions.
This makes me think of Python 2 vs 3 benchmarks for some reason, although I'm sure I'm not remembering very well.
I assume there's plenty of scope for ES6 to improve and (hopefully?) surpass ES5 in performance in many, if not all, areas?
What would be causing the large slowdowns in ES6 here? Is this a case of having features that make JS nicer to code in, but giving up some performance? Or is this more down to immature compilers not yet optimising ES6 as substantially?
ES6 is the native implementation in the browser or JS-engine you are using, indeed.
Green is probably chosen because the same performance is nice to have with this (subjectively) better syntax. The darker shade is probably chosen because more performance is nice, as it pushes you to the newer (subjectively better) syntax.
Note, when it says "1.6x faster" what is really means is "1.6 as fast" or "60% faster", or really "operates at 1.6x the speed of the baseline", not "1.6x increase in speed".
For example, when looking at the data for Chrome 48's "arrow" tests [1], the baseline number is 57,858,016 and the traceur number is 91,556,806, which is 158.2% of the baseline, but is reported as "1.6x faster".
I know this seems like semantics, but "1.0x faster" sounds a lot like "twice as fast". (In this case, "1.0x faster" is reported as "identical" [2]).
They did say "1.6x faster", not using the term "speedup" (which is terrible nomenclature too, but that's not the point).
To say that something that's no faster is identical to something that's "1x faster" is a terrible confusion of thought.
It's reasonable to expect that, for example, "1x faster" would mean "for a baseline of 1 operation per second, it's faster by 1 operation per second, i.e. a total of 2 operations per second."
EDIT: Oh god, that's even before I saw the "1.2x slower" entries…
I agree based on my experience reading these types of comparisons. I've never thought there was any ambiguity to saying "1.6 times faster." It means performance_1 = performance_0 * 1.6.
>Note, when it says "1.6x faster" what is really means is "1.6 as fast" or "60% faster", or really "operates at 1.6x the speed of the baseline", not "1.6x increase in speed".
Isn't this always the case with such comparisons? One is meant to multiple initialValue by e.g. 1.6 (hence the "x", for multiple), not calculate initialValue + 1.6*initialValue.
No, the word faster means an increase by that amount. 100% faster means twice as fast. 1x faster means twice as fast. 100% as fast means no change. 1x as fast means no change.
I'm not sure I've ever seen anything described as being '1x' faster, without any fractional part. It's such an uncommon usage, I really don't think you can categorically declare that '1x faster means twice as fast'.
Not really, that doesn't make sense. It's "0.5x the speed of b". Saying "faster" when it's really actually slower is just confusing and ungrammatical. Likewise saying "1x faster" when it's actually the same speed is gratuitously confusing.
What doesn't make sense? Saying "A is 50% faster than B" is equivalent to "A is 150% as fast as B". Same goes for "A is 0.5x faster than B" which is equivalent to "A is 1.5x as fast as B".
There is a very distinct difference between "faster than" and "as fast as" which should be very clear now.
You're 100% right (2x right?), but I think this is one of those situations like "bimonthly", where people usually intend it to mean twice a month instead of every two months.
It's completely ambiguous with potentially significant costs for misunderstanding, but people refuse to make themselves clear, just because.
For "A was 50% faster than B", it could very easily be B which is faster -- and A is half its speed.
What I meant to say in my parent comment is not the use of "faster" vs "as fast" in English, but that usually in those benchmarks "Nx" means N times the speed of the baseline.
Again, I wasn't talking about what's it in "proper english" -- but how it's used in benchmarks.
Kevin Decker, who did the site in TFA, is a native english speaker, and yet he writes "1.5x faster" to mean baseline1.5 (and not baseline + 1.5baseline).
So clearly it's not so "unambiguous" whether you can use faster in this way or not. And in fact, I've never seen any of those benchmarks where "Nx" doesn't mean the final value is N*baseline.
This confused the heck out of me as a kid. Saying something is 50% faster means the same as being 150% faster. Or 100% faster could also be 200%, depending on subtle language choices.
Correct, at the extremes, the numbers don't matter. But where it says "1.2x slower", it really means "84% of baseline" [1], which might be a bit misleading.
The "commentariat" charge is legitimate, which is why there is a field known as "mathematics" which is unambiguous. 1.6x by anybody's non-commentariat definition is 60% faster, as anybody who has ever even scratched the surface of any mathematical, engineering, scientific, statistical or indeed, computer science discipline knows (though this is perhaps unknown to the Creative Suite crowd). This comment is completely bogus, with apologies if you don't understand. But you really should know what you don't know before posting such garbage.
Also you're wrong. "__ faster" implies addition in many/most cases. Since you know what 1.6x is by itself, I'll let you use your math knowledge to calculate x + 1.6x.
This deopt led to a several orders of magnitude slowdown compare to a simple `for` loop in Google Chrome, forcing use to abandon the `for of` construction.
I suspect that the benchmark shown on this page does not expose this kind of behavior because it doesn't iterate enough for the JIT to kick in. If it's the case, it means that these values reflects only the behavior of the cold, interpreted code, and not the hot one. (which is quite sad for a performance benchmark, because the performance matter only for the former …)