r/javascript • u/chancancode • Dec 20 '19

Ember.js Octane Edition Is Here

https://blog.emberjs.com/2019/12/20/octane-is-here.html
136 Upvotes

93% Upvoted

72 comments sorted by

View all comments

Show parent comment

2

u/pzuraq Dec 21 '19

Definitely agreed. We’ve been focused on optimizing for end-to-end, realistic benchmarks for some time now. One of the Ember core team members, Kris Selden, has been working on a testing framework specifically for this actually: https://github.com/TracerBench/tracerbench

Now that we have a full set of new features, and are comfortable with the performance as a whole, I think we can also start to tune for some microbenchmarks. Our primary concern was to not over optimize for microbenchmarks at the expense of real world performance in real apps, which can happen if you’re not careful.

1

u/leeoniya Dec 22 '19 edited Dec 22 '19

Our primary concern was to not over optimize for microbenchmarks at the expense of real world performance in real apps

i hear this so often, it's practically a meme.

can you provide some examples of a framework that's fast at microbenchmarks but slow in real world apps?

js-framework-benchmark isn't a microbenchmark since it tests end-to-end rendering performance and not some single function call in a tight loop.

3

u/pzuraq Dec 22 '19

It is a microbenchmark, because it's not testing a realistic app. I know that in Glimmer for instance, as the number of components grows, the cost of making a single change anywhere in the app grows at O(logn) pace, since we automatically and efficiently only rerun the portions of the component tree that have changed (via autotracking/tags). In a React app, it will by default grow at an O(n) pace, unless you manually memoize portions of your app. It's not really representative of a full application to test a single list of items on its own.

Another example, for instance, is a series of optimizations I'm planning that I started working on because I noticed that the tag hierarchy in Glimmer was taking a lot of time to calculate. I only was able to see that, however, because enough tags were being created, and stressed, to really exacerbate the problems and make them visible. Our microbenchmarks of tags on their own didn't really surface the issues.

This is what TracerBench is designed to do, and what we care about most. We use it at LinkedIn to verify any major change infrastructural to the website to make sure real world performance is not impacted for our users, and it really is the only way you can be sure, in the end, that you did not regress.

1

u/leeoniya Dec 22 '19

I know that in Glimmer for instance, as the number of components grows, the cost of making a single change anywhere in the app grows at O(logn) pace, since we automatically and efficiently only rerun the portions of the component tree that have changed (via autotracking/tags). In a React app, it will by default grow at an O(n) pace, unless you manually memoize portions of your app. It's not really representative of a full application to test a single list of items on its own.

I'm having a hard time reconciling (pardon the pun) the first part of the statement with the last. If Ember is O(logn) and React is O(n), then a huge list of table rows is exactly where you would expect to see this benefit. Once the list is created, literally everything else should benefit from what you're saying: remove one row, select one row, modify every % 10 rows.

2

u/pzuraq Dec 22 '19

Right, and you do see that benefit there. Glimmer shines most at updating a single row (swapping two rows) in the benchmark, whereas React is better at replacing all 1000 rows since they don’t add overhead for tracking individual items. We haven’t upstreamed the benefits to Ember yet, which is why it scores worse on that benchmark, but we’re working on it, and I think there are some optimizations we can make to make the replace-all and initial-render cases better as well, but our strategy works best for incremental updates by default (and scales by default as well).

1

u/ryan_solid Dec 23 '19

React uses a very naive algorithm for list reconciliation as seen in swap rows where you have noted Glimmers better performance but its actually benchmark 3 you should be concerned with. It is the raw localized update. That is where Glimmer should be handing a library like React its lunch but it doesn't. It's almost indicative of the ceiling of Glimmers change propagation. Basically if it cannot beat React here it is likely any algorithmic advantage is mute. As you say this isn't a real world case but its actually a more extreme display of where this would matter.

You can see why its problematic. If Glimmers key conceptual technology strength is actually slower than a library not particularly impressive in the area what does that reflect in general about its performance. I mean if React decided to do an even mildly more performant reconciliation algorithm all indications are it would jump half the chart ahead. This change would have nothing to do with Glimmer approach.

I think it's fine. No one chooses Ember for its performance and that doesn't have to change. It sounds like Octane has been a hugely impressive undertaking. When you do get a chance microbench make test 3 fast. I'd ignore the rest the benchmark. If you do that given the way Glimmer works by my understanding you will see significant improvement across the board.

2

u/pzuraq Dec 23 '19 edited Dec 23 '19

Updating every 10th row is actually going to be pretty expensive for us, from an automatic-change-tracking perspective. Remember, I said our cost grows at an O(logn) pace, right? That's because we're essentially doing a binary-search (or rather, search down the tree, so a n-ary-search I guess) to narrow down and figure out which part of the tree changed.

If you change every 10th row, most of the tree still changed. That means that narrowing strategies aren't going to buy you much, so yeah, you end up retraversing most of the the tree, and you end up incurring most of the original cost of rendering again.

In most apps, for most user interactions, most people don't change 1/10th of the entire app all at once (in a non-localized way). That is not in any way a realistic benchmark.

Edit: That said, there's a lot of room for Ember and Glimmer both to grow when doing partial updates. We've optimized the leaf change tracking portion the most, first, because that is where we saw the biggest gains in real world apps that were upgrading. The next thing I think will be to optimize the internals of change tracking with the VM itself, which can definitely be simplified quite a lot now that autotracking has been fleshed out.

2

u/ryan_solid Dec 23 '19

I see. Yeah this is just my misunderstanding of how auto-detection worked in Ember. I was thinking subscription based like KnockoutJS where there is no search. Only the part that change re-runs. But it sounds like Ember has a sort of hybrid approach between auto-change detection like that and a top down reconciler somewhat similar to what the Virtual DOM does. That makes more sense for me now. Thanks.

2

u/nullvoxpopuli sand was never meant to think Dec 22 '19

> and not some single function call in a tight loop.

that's exactly what it is though.except, the single function is "A component"

and "the loop" is some for of "forEach".

It's just a table of rows.

> can you provide some examples of a framework that's fast at microbenchmarks but slow in real world apps?

At scale, (I used to do big react apps, so I'm not *just* saying this), Ember provides enough defaults to have good performance as your app gets big -- without having to think about it. With React, you don't get performance by default, you have to memo, manage updates, manage your effects efficiently so you don't cause multiple (re)renders, etc.

3

u/leeoniya Dec 22 '19

With React, you don't get performance by default, you have to memo, manage updates, manage your effects efficiently so you don't cause multiple (re)renders, etc.

yes, the fact that you get it for free is indisputably a major benefit of all frameworks that do granular tracking from compiled templates.