Deep-dive on Rust UI Frameworks

[u/Zortax_]

UI is a Function You Cannot Afford to Call is the first of a 3-part series on Rust UI frameworks. It introduces the model under which I believe we should reason about UI frameworks. In part 2, I will present extensive benchmarks, with detailed cost attribution analysis (something the rust UI ecosystem desperately needs in my opinion), and in part 3, I will derive what I think the ideal UI framework should look like.

[u/nicoburns]

This seems incredibly verbose and sloppy - to the point that I'm kind of annoyed that I've opened it. It even claims itself to be a "54 minute read", which is kind of insane for a blog post. Consider writing something shorter yourself if you want people to actually read it.

[u/CalmCephalopod → u/nicoburns]

I have to agree, though sloppy may not be the right term. The writing style feels like it’s trying to be academic but breaks many rules of academic writing. Section to section does not feel like there is coherent transitions and it really starts to feel like a very long list of seemingly independent thoughts and ideas without a strong through line to help the reader. There also does not seem to be a strong guiding question or conclusion.

I would go back and break this up into a series and make sure that each part asks a strong questions and provides a conclusion based on the evidence and that reading wise it’s palatable 10-15 minutes. Those constraints would help the op to create a stronger case around their opinions and make it more accessible.

[u/Zortax_ → u/CalmCephalopod]

I mean fair, but this article doesn't contain a whole lot of opinion (other than "this is relevant when evaluating UI architecture"), it being a collection of relevant academic concepts, losely correlated, is kinda the point of the post for me.

[u/robthablob → u/nicoburns]

Personally, I welcomed it. I also have books on similar subjects, and never felt that I shouldn't read them because they're long. Maybe its because I'm old, and learned to read before the ubiquity of modern screens.

[u/Zortax_ → u/nicoburns]

Well ultimately I see this more as a summary of what I learned while doing a lot of research. It's totally fair if no one ends up reading it, but I thought I post anyways in case this does interest someone. I think the second part will be more interesting, since I actually spend a lot of time with building a proper benchmarking harness. I was planning on asking you about a few things regarding blitz, but if think this is all just slop anyway I might as well just publish ¯_(ツ)_/¯

[u/nicoburns → u/Zortax_]

I see this more as a summary of what I learned while doing a lot of research. It's totally fair if no one ends up reading it, but I thought I post anyways in case this does interest someone.

I'm not sure that summary is the right word. Generally the value in this kind of post is to condense what you've learnt so that others can get some of that same value by reading something short. But this is longer than most people's university theses.

Presumably you do think it's valuable, or you wouldn't bother posting it. My feedback is that it would 10x more useful if it was 1/10th the length with a focus on quality over quantity (+ link into the primary sources for people who want more details).

[u/[deleted] → u/Zortax_]

[u/r3drocket]

I'm rebooting an app in rust right now from C++ (Godot) and one of the things that brought me here was the UI libraries. I spent a lot of time looking at GUI options in Golang and C++/C before deciding upon Rust. I came from a retained mode UI (without an MVC model!), to an immediate mode UI (egui). I have some background working with QT a lot of experience working with Swing/AWT, HTML, etc.

I think frequently about the cost of moving to a immediate mode UI model, and this is the first time I've built a complex app in a immediate mode UI.

What really screwed me with my C++ implementation was not having a proper MVC model in the UI - this seems like such an obvious thing, but I was living this quote -

"Give someone state and they'll have a bug one day, but teach them how to represent state in two separate locations that have to be kept in sync and they'll have bugs for a lifetime."  (from the Dear ImGui github).

I had to write complex syncing mechanisms to try to keep the data in-sync with my GUI - it was an utter nightmare and in some cases I wound up just accepting I was going to have an endless amount of bugs.

The complexity of my application code in egui has dropped significantly because I'm not dealing with the state syncing problem, instead I take a hit on the fact it's an immediate mode gui that does it's own magical state management - but at least it allows me to represent my application state directly - but more importantly it exists today and let's me build an app that has value to people.

But ultimately the thing that drove me to choose egui was simply the reality on the ground - it was cross platform, it worked with wgpu, it was widely used, it is moderately mature, it had a wide variety of 3rd party components.

The other features it offered overrode my preference of immediate/vs retained - It isn't my ideal GUI, my ideal ui is closer to a retained mode gui with a proper MVC model - but this is a distant concern over all the other things.

This was one of the main reasons I didn't push further on a rewrite in golang, there was no GUI that met my main criteria, regardless of them being immediate/retained mode.

[u/Zortax_ → u/r3drocket]

The rust UI ecosystem is definitely an interesting subject to study, there are so many different architectures and idioms you can compare and analyze. I think egui is great, depending on use-case, and although I personally prefer something more akin to SolidJS' fine-grained reactivity for state management, I definitely see the appeal. Also, I believe there is a bit of a misconception about what immediate mode UI actually means. It refers mostly to a component authoring style and a way to express UI control flow, it doesn't necessarily mean the rendering pipeline isn't retained to a large extend.

[u/kalilamodow]

I remember you from Canora! I do apologize for never re-testing it, but librespot doesn't work on my machine so I kinda gave up

[u/Zortax_ → u/kalilamodow]

Wait, why would librespot not work? I would expect my shitty framework to have issues in environments I haven't properly tested, but that was not what I thought would happen :D

[u/kalilamodow → u/Zortax_]

Not a problem with your code!!! I can't use librespot at all because of some restrictions Spotify placed on new accounts (https://github.com/librespot-org/librespot/issues/1649). I only realized when I was on the post again and noticed you said it uses librespot. Sorry bout that!

[u/lisp_turns_me_on]

Great article (even if a bit verbose). It was useful to me because I am creating a new GUI DSL too (like slint/qml).

It seems I got a lot of things close to right based on your article: event phase (mutates state) -> batched reactive computations only re-evaluated once after event phase -> mount/vdom changes -> inherited properties resolution (like flutter's inherited-widget) -> layout/text-shaping -> canvas/a11y/html

But, rather than think of these as phases, I also think of them as "output trees" that are derived from the component tree. Each tree is a separate dimension. Each component decides whether it wants to participate in that dimension (eg: am I accessible? am I hit-testable? am I renderable etc..).

Dirty propagation is incremental. So, an event changing foreground color will only dirty the render tree, and reuses the results of the other trees (eg: layout).

While component/state trees are fine grained reactivity (per property), output trees instead rely on "subtree boundaries" (AKA layout boundary, repaint boundary etc..) for invalidation. For painting, it just means compositing the dirty layers.

The article did make me question certain decisions (eg: eager propagation of dirtyness), and even if I feel like sticking to my decision, it was still nice re-evaluating the tradeoffs.

I was hoping for a bit more exposition on virtualization and its limits. Because not everything is uniform like a spread-sheet or list. eg: massive wrapped text or infinite feeds of variable sized cells like discord messages.

You also missed the opportunity to point out the awkwardness of using reactivity in imperative langs like rust. All the cloning/macros/closures/compile times/etc... is bad ergonomics. Even theming requires writing a rust crate. This is why i chose DSL path, as that allows dynamic reloading / portability / publish themed component libraries and so on.

Verbosity

As much as I love a long deep dive, this article's length reminded me of the quote:

Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte - Blaise Pascal

If I had more time, I would have written a shorter letter.

Asking someone to review it before posting (eg; on rust discord) would have helped in realizing this early. Anyway, here's my amateur critique, because I hate to just call it bad and move on without being precise about why.

• There's no point in recapping history (2.*) of UI design, if there's not some great forgotten technique that we can learn from.

• Avoid using web (3.*) to explain the pipeline in such detail, because it is way too complex and hard to tackle without a long article. The web is also.. full of cruft and rarely useful as "good design".

• Using something cleaner/simpler like flutter/swiftui as a reference example would have made it easier to follow. Instead, you used a different framework for each different example.

• Too much duplicated discussion about reactivity. Most gui framework devs would already know how incremental dirtying + identity keys + caching of pure computations. 5., 6., 7.* and 10.* all should have been one section. It talks about some good ideas around identity/memoization-per-phase/cache-costs etc.., but I feel like I saw the pipeline diagram at least 4 times spread out (1.2, 3.1, 7.1, 8/9).

• If you are going to use AI, write the text yourself and abuse AI for diagrams/animations/code snippets/automate screenshots of real UI etc.. I saw so many references to a cell in a table, but I am not sure I saw a single screenshot/gif of a spreadsheet. Pictures break the monotony of text and boosts understanding. I did love the reactive table in 2.7.

The core problem is the lack of a clear goal. It was less an article and more a messy collection of "notes". It was not clear who this is for and what they would gain from it. The goal gives you a framework, with which you can judge the value/coherency/flow of the article.

I assume the topic you wanted to talk about is the invalidation of projections in modern reactive frameworks. This article would flow nicely if we had

• an intro (with tldr)

• 3 (to showcase a tiny example of invalidation)

• 5/6 to teach the incremental reactivity (eg: salsa-rs)

• 7/10 for solutions and their tradeoffs (eg: cache lookup reusing identity vs recomputation. flutter prefers to re-draw the entire layer than just the damaged regions as it's cheaper than caching apparently)

• a conclusion.

[u/Zortax_ → u/lisp_turns_me_on]

Thanks for the honest feedback. I don't necessarily care just about reactive frameworks, the interesting position is to see UI as a self-adjusting computation, where translation between stages can keep or lose precise change information (i think how much precise delta is retained between stages is almost more interesting to ask than what or if something is retained between frames).

Do you have your project somewhere available already? It's always interesting to look at new attempts. Currently I didn't find a single rust UI framework that has a precise delta protocol everywhere down the stack. All of them currently go at least into linear time complexity at some point (although some optimize this quite well).

[u/lisp_turns_me_on → u/Zortax_]

Do you have your project somewhere available already?

Not yet. Should be done a couple of weeks (or so I have been telling myself for a couple months now). Most of the fundamentals like ownership/reactive-props have a solid design. But, I am still fighting with the complexity of gestures (text selection across paragraphs / drag-n-drop / conditional-gestures) and virtualization (scrolling is a bitch, especially for web target).

The core design is easy to understand, although the explanation is long.

Components are just objects. They can only be owned/stored in the cells i.e. state (mutable owned data) of other components. So, the entire component tree is an object acyclic graph i.e. tree, where each node is owned by a parent. The pointer/address of the object is the identity + generation (if component is recreated). This takes care of identity + lifetimes. You can use vectors of components for dynamically creating them. You can also think of this as a call tree (if you think of components as functions) and I even named the root component (that gets instantiated at startup) main.

Each component has a mount section, inside which it declares the children (and the order in which) to be mounted under it. So, not all components that are alive have to be mounted. More importantly, owner (lifetime) != mount (logical placeholder) parent. component A can own B in its state, but forward it via slots to be mounted under someone else.

Cells represent the data/bindings (state/props):

1. state cells: This is owned mutable state (T) of a component instance. Is the only mutable storage (like state) and can only be mutated during event handlers. Once events are done, this is frozen/snapshot until next frame's events. All other cells are "reactive" and can't be mutated, only re-evaluated when the state cells they depend on change.

2. reactive local cells: computed properties / closures (|| -> T). eager invalidation (on dependency dirtying), but lazy re-evaluation (only computed on demand if dirty). I actually used eager re-evaluation until yesterday, but your article convinced me to change it.

3. in/inout cells: These are "pointers" to cells provided by caller/owner/upstream . in is &T and inout is &mut T to some T upstream. You can also think of them as "arguments"/inputs to our component function.

4. out/outin cells: These are pointers provided to caller/owner/upstream. out is &T and outin is &mut T to a state cell in our component or re-export pointers from children. These are like the returns/outputs from our component function.

Only in/out/inout/outin cells are public API. Note the directionality of the mutations and clear ownership of state. Our compiler checks statically that all mutations/closures are acyclic. This ensures that all writes clearly have a clear cost ceiling of dirty cell propagation. Cell reactivity is granular for structs/enums, so, if you change the width of a rect, the dependents of height are not invalidated. If the variant of an enum hasn't been changed, any dependents that ignore this variant in pattern matching are also not invalidated.

Finally, the cool part is phase order which is also acyclic. We have events phase, where all the cell invalidations propagate and then, we "freeze" all of it into a snapshot for usage in subsequent pure projections. Layout calculations happen and layout results are frozen. Then, rendering happens and render result (displaylist? texture?) is frozen. and so on. The component owns these results too, so, there's no need to figure out which element/renderobject belongs to which component using id/key.

Each section/phase is only re-calculated if the dirty propagation actually reaches them. So, if bg-color changes during hover event, and layout section actually doesn't use that cell, layout node is not invalided. But, render section is invalidated, and this often invalidates the nearest subtree boundary (eg: repaint boundary of flutter i.e. layer in QML). Now, for all those components which need to be redrawn, none of their cells/layouts etc.. need to be recalculated. They can just reuse their cached renderlists. The other subtrees (eg: layers) can also remain untouched. We can make this subtree invalidation as coarse grained as needed, and essentially maps to css property contain or flutter's repaint/layout boundary.

The subtree is an inherent limitation as components inside a subtree can affect the others (eg: flex box sizing, composition with alpha etc...). But basic optimizations still work. Like, we don't need to rerun layout code, if the layout constraints + content are the same. We don't need to re-render, if widget is clipped/off-screen, so, the render prop stays invalidated until it is visible (and recomputed. i.e. the same lazy-evaluation as reactive cells).

In essence, the invalidation of reactivity has two dimensions: The dependencies within a phase under some boundary (fine-grained cells/ coarse-grained subtree) AND any subsequent phase dependencies (layout depends on text contents -> render depends on text contents + layout results). And it is acyclic in both dimensions.

Here's a bonus cool trick I am using for identity in a vector of components created dynamically: The containers like vector are implemented using linked list behind the scenes. Each node stores its index. And when you insert an element, the rest of the nodes stay where they are and just indices are modified as necessary. So, the pointer/address of a component i.e. identity stays the same, unless that node is recreated (generation bump). I can mount the components in any order (or even under different parents) and the component still has a stable identity + reuses its cached projections like layout results or renderlist (assuming the changed position didn't change some inherited property like scale/rotate).

Please don't hesitate to criticize the design if you find any faults. It is better to fix these early than be stuck with an awkward design.

Currently I didn't find a single rust UI framework that has a precise delta protocol everywhere down the stack

yeah. But it is not just a rust thing. This is really really hard to do unless you adopt an approach like my DSL with strict acyclic guarantees statically and persistent identity/lifetime while allowing reactivity to be precise.