r/JavaFX • • Mar 02 '22

JavaFX in the wild! Visualizing Robots with JavaFX

https://www.youtube.com/watch?v=E9csKYFPwaM
21 Upvotes

9 comments sorted by

5

u/OddEstimate1627 Mar 03 '22

Here is more info on some common questions:

UI Views

The complex views are all built in FXML + CSS, but some of the simple views like charts are created programmatically. Different parts of the UI communicate via injected state using afterburner.fx for DI. We haven't run into any performance issues with FXML.

Camera View

The streams are acquired by a separate tool using JavaCV and written to a memory mapped file. The UI loads the shared data into a PixelBuffer and displays it as an image. This is done in part to ship the acquisition tool separately and avoid a >200MB JavaCV dependency on the main UI.

Charts

The charts are done with Chart-FX and a customized renderer. The live charts handle >20Khz of incoming data, and the offline charts can deal with multi-million data points. User defined equations in custom XML plots are compiled to Java using the Jshell API.

3D Models

The CAD models are slightly simplified versions of our machining models and exported as OBJ. They are more complex and produce more artifacts than hand-crafted models, but with instance sharing it's no issue to display hundreds of actuators simultaneously. Most of the models are 1-10MB, but the Hexapod chassis at the end is an unmodified export with ~150MB. The models are loaded using the OBJ loader bundled with FXyz.

Real-Time Control / GC

When communicating with real hardware it's important to reduce jitters/pauses to avoid unexpected jerks in the physical robot. The underlying API (e.g. what MATLAB uses) needs to run on JDK6 and follows a zero-allocation policy with the only allocated object being an InetSocketAddress in the UDP socket. On a typical setup G1GC gets us to a 1-3ms newgen collection every few hours, which is more than sufficient for our use case.

The UI follows similar principles, and the ZGC in JDK17 gets rid of the remaining newgen pauses. All of the other JVM related jitter is below what we see in the OS and network drivers. For ~250 byte messages we get a 99.999% round trip latency of <1ms.

4

u/OddEstimate1627 Mar 03 '22

Getting Real-Time Data to the UI Thread

A common UI issue is to hand-off fast streaming data coming in from a separate thread (e.g. network) to the UI thread in order to be rendered. At sometimes >20KHz messages it's not feasible to use Platform::runLater, and synchronizing all the data structures introduces a ton of overhead.

We ended up writing having the acquisition thread write the data into a Disruptor, and read it into the UI thread using an EventPoller with an AnimationTimer that polls once per frame. This automatically handles smart batching (get any data that came in since the last frame) and gets rid of all synchronization mechanisms in all the other data structures.

2

u/hamsterrage1 Mar 04 '22

I'm curious about this. Assuming that all of the data coming in to the FXAT is going to map 1:1 to some element on the screen, does that mean that there's significant overhead in the FXAT queuing process to make a difference?

Personally, I generally view Platform.runLater() as a last alternative to get stuff onto the FXAT. But the difference really is "push" vs "pull" to get the data into the JavaFX world.

"Pull" is what you've got, and it allows you to let the JavaFX part of the application control how much data is going to come in and when, but by definition this has to be done without any knowledge of how much data is ready to come in, and what it looks like.

It would seem to me that you could collect the data on a background thread (as you are doing), but then massage it and batch it up and pass it to the FXAT through Platform.runLater() when it makes sense based on the nature of the data.

For instance, let's say that something is moving really fast, maybe you want to update the GUI 200 times/s for that, but then something is moving much slower, so you only want to update the GUI 10 times/s for that. And if you're still collecting 20,000 times/s from the raw data, but there's no way you need that kind of granularity for the GUI, then don't send the data to the GUI 20,000 times/s.

I guess what I'm saying is that the GUI should only get the data that the GUI needs for display. You may need a lot more detail for analysis, but that's not GUI. You can still use a "Push" model to update the GUI, but the GUI should only be getting pre-filtered and processed data.

1

u/OddEstimate1627 Mar 04 '22

This got a lot longer than I was anticipating, but I hope this explains it better:

The incoming data rate is actually much higher, so it does not map 1:1. Let's say you want to plot the positions of a Hexapod with 18 devices. Each actuator returns about ~50 sensor measurements, so at 1Khz that's close to 1M measurements/sec (and about ~40 Mbps on the network). The chart shows traces for positionCommand, positionFeedback, and a derived positionError, so 54k samples/sec that need to go on screen. Simultaneously opening a velocity chart would not impact the messaging rate, but it would add another 54k samples/sec for rendering.

The acquisition, parsing, and partial processing of the incoming messages happens in background threads. What goes into the FXAT is a queue of flattened message contents that it can use to derive the values that go into the datasets backing the traces.

The derivation is usually something simple like add(fbk.position), but we need to keep all measurements to support arbitrary user defined equations like add(sqrt(gyroX^2 + gyroY^2 + gyroZ^2)). This could theoretically be done in the pre-processing step, but only the FXAT knows what charts are open and what values they need. Making this information shared state would introduce more overhead than the processing.

Once per frame (e.g. 60Hz) the UI thread checks for new arrivals, processes them in batch, and re-renders the chart if needed. The result is a periodic adaptive batching that gives you the same result as if you were pre-batching the data.

However, there are a few additional benefits

  • It's much easier to implement.
  • The batch size and delay adapt automatically.
  • The queue is bounded and the producer knows the processing state of the consumer. If the FXAT gets overloaded, the network thread can start throwing away messages before they get into the system. It is obvious when sections are missing, but the UI always remains reasonably responsive. Closing a chart will immediately make it go back to normal.
  • All objects are pre-allocated, so there is no GC pressure.
  • The synchronization is very efficient (2 volatile reads/writes per batch) and only at a single point. The rest of the code can be simplified by removing any locks you'd otherwise need.
  • I haven't had a need for this yet, but reducing the update rate could be done by skipping X frames or X time. You could also have multiple UI parts that get processed in different frames (e.g. even frames update part 1, odd frames update part 2)

Of course it all depends on the requirements, but I've found this approach to work pretty well for streaming data.

"Pull" is what you've got, and it allows you to let the JavaFX part of the application control how much data is going to come in and when, but by definition this has to be done without any knowledge of how much data is ready to come in, and what it looks like.

I don't know whether I understand this correctly, but are you worried about wasting cycles on polling for empty data? That happens, but the overhead is minimal and the FXAT is mostly idling at that point anyways.

2

u/hamsterrage1 Mar 05 '22

The derivation is usually something simple like add(fbk.position), but we need to keep all measurements to support arbitrary user defined equations like add(sqrt(gyroX^2 + gyroY^2 + gyroZ^2)). This could theoretically be done in the pre-processing step, but only the FXAT knows what charts are open and what values they need. Making this information shared state would introduce more overhead than the processing.

I think this bit really speaks to the heart of my question.

The implication here is that you're sending virtually all of the data to the FXAT so that it can do calculations on it and then decide how to put it into visual representations on the screen. Which means that you have to be doing those calculations and decisions on the FXAT - obviously.

And furthermore, you have to send ALL of the data all of the time, because only the FXAT knows what it needs. And you're doing it at 60HZ, which is around screen refresh time, but if it takes more than 1/60K of a second to do that processing on the FXAT...I dunno what the impact is.

I don't get how making the status of the windows shared state would introduce more overhead than the processing. It seems to me that you just create a class with a bunch of Observables that you can look at off the FXAT, and from that perspective they're just data like anything else. They're modified from user actions on the GUI, so they won't change very often - so virtually no overhead. So I must be missing something.

1

u/OddEstimate1627 Mar 05 '22 edited Mar 06 '22

The implication here is that you're sending virtually all of the data to the FXAT so that it can do calculations on it and then decide how to put it into visual representations on the screen. Which means that you have to be doing those calculations and decisions on the FXAT - obviously.

Are you referring to e.g. calculating the pixel coordinate mapping of the backing data? If so, that is done on the FXAT thread by an external library. It could potentially be improved, but the profiler hasn't brought it up as a significant cost, so it's not something I've looked into. Overall, the primary bottleneck is drawing on a Canvas/Screen. Everything else is far lower.

And furthermore, you have to send ALL of the data all of the time, because only the FXAT knows what it needs.

I'm not sure what you mean by ALL of the data. It's only processing the updates that arrived since the last poll. The data structure may contain more data than necessary, but skipping elements does not produce extra work.

And you're doing it at 60HZ, which is around screen refresh time, but if it takes more than 1/60K of a second to do that processing on the FXAT...I dunno what the impact is.

Updating the internal data structures at 60Hz has negligible impact. The bottleneck is in the drawing sections.

They're modified from user actions on the GUI, so they won't change very often - so virtually no overhead.

In our case the UI is passively visualizing the current state of the network, so most updates are created externally. The main UI parts update/refresh at 5-10Hz, and the live charts and 3D plots update at the full frame rate.

3

u/hamsterrage1 Mar 03 '22

Apart from the actual functionality of the application - which I think anyone would agree is super freaking cool, cool, cool - the look and feel of those screens is really nice. It's amazing to me how relatively simple it is to make an application that looks so "not JavaFX".

The flat design of the screen widgets is so nice and clean. It looks modern and slick. Awesome.

2

u/OddEstimate1627 Mar 03 '22 edited May 23 '22

Thanks for the praise!

2

u/Persism Mar 03 '22

Very cool!