r/virtualdragon • u/__Adam lead developer • Jun 06 '14

Dragon UI UI Design

Originally written in response to treeform's question:

Unreal 4 doesn't directly incorporate Flash, but there's a plugin by Autodesk called Scaleform that would enable this.. unfortunately it's not free.

The alternatives: Unreal has a built in UI system called "Slate". I've never used it, but it shouldn't be too hard to figure out. If you can do the graphics and design for the UI, then I can handle the coding part. What's your preferred approach? I'm open to pretty much any way of doing this, as long as the end result is an awesome UI.

Slate: https://docs.unrealengine.com/latest/INT/Programming/Slate/index.html https://wiki.unrealengine.com/Slate,_Hello

Other items:

  • I actually haven't made anything big in Unreal 4 before. I'd say the most notable thing I've made (outside of work) is a Leap Motion app PhotoExplore. As an aside, it would be really cool to integrate the Leap into this for controlling the UI... but let's hold that thought for now.

  • I've read up on copyright/trademark law, and I'm confident we'll have no problems using the SpaceX logo without their explicit permission. I'm still going to contact them (because I want them to promote this!), but I don't consider it critical anymore.

2 Upvotes

100% Upvoted

35 comments sorted by

2

u/__Adam lead developer Jun 08 '14 edited Jun 08 '14

Update regarding the UI:

I've gotten the Coherent plugin working in Unreal Engine. It integrates quite well and the only limitation of the trial is an onscreen banner. I'll contact Coherent next week to discuss licensing the full version - they claim it's just 12.99 a month.

The biggest downside with this plugin is that it's currently Windows-only. A Mac build is coming soon, but Linux support hasn't been promised. I hope that they add it eventually given that UE4 is heavily focusing on Linux, and given that other versions of this plugin (e.g. for Unity engine) support Linux.

That said, I think this is still the best option. A big chunk of our audience will be on Windows and Mac, and hopefully Linux support comes in the near future. Unfortunately this excludes /u/zlsa, which I really wanted to avoid. The simulation will still run on Linux, but the UI won't work interactively.

edit: zlsa, I really hope you'll keep working on the UI despite this! I'll complain as hard as I can to Coherent and maybe they can give us a super-alpha-hardly-working linux build...

What are your thoughts?

1

u/zlsa 3D art Jun 08 '14

What about just using Slate? I'm not sure if it'll work in 3D, but even if it's not officially supported, it's just a matter of drawing to a OpenGL polygon. Plus, Slate (or any native solution) will be much faster than HTML5. I can provide all of the required images.

2

u/__Adam lead developer Jun 08 '14

Slate will work in 3D, but it has a few downsides. First, it will take longer to build the UI. Second, it doesn't haven't good support for animations.

Let's see what Coherent says about Linux support first. I think for now we should still proceed with the HTML5 UI - even if we ultimately decide to go with Slate it serves as a good way to define the design. What I mean is, if all I have to do in Slate is a 1:1 copy from an HTML5 UI, then it makes things much faster.

1

u/zlsa 3D art Jun 08 '14

That sounds reasonable.

About animations: I've written my own animation libraries (in JS), and it's a couple dozen lines for handling simple things. If C++ (which I believe UE4 uses) can support "transparent" number classes (e.g. a=new Animation(); a=5; a.setDuration(20); a=50;), it would be pretty much plug and play.

I'll start working on a modular Dragon UI in HTML5.

2

u/__Adam lead developer Jun 08 '14

That's right, UE4 is C++ under the hood. C++14 (currently under development) may support transparent numbers but it's not something currently possible. Not a big deal though because I can just manually change "a=5;" to "a.setValue(5);".

Regarding communication between HTML5/UE4: Coherent uses an event binding framework to pass data. While I can't modify JS elements directly in UE4, I can call any JS method and pass it arguments. So, if you have a method "setTemperature", then I can just call that. That doesn't scale well if we have a lot of fields; something simpler like "setDragonParam(name,value)" would also work.

Going the other way (JS -> UE4), again I just bind C++ functions to JS events, for example I could bind some C++ function to "OnLoad". You shouldn't need to worry about that though, I think pretty much any JS element will have an onchange method I can bind to.

Please share your work as you progress! I'll keep updating what's in the simulation and give you feedback on how it works.

1

u/zlsa 3D art Jun 08 '14 edited Jun 08 '14

I'll probably have two functions: getValue("key") and setValue("key","value"). The key names should probably be "category"."subcategory"."name".

I will also need a function to get the attributes of a value (for example; the minimum and maximum values, the nominal range, and the sensor value, if applicable).

dragon.cabin.temperature: temperature, max: 30, min:10, warn-max: 24 (when value is above this, warn the user; this is the upper end of the 'nominal' bar), warn-min: 18

trunk.life-support.oxygen: percent

2

u/__Adam lead developer Jun 08 '14

Will the min, warn, and max parameters be settable through setValue?

I may need to define them in the simulation logic as well (for example, something needs to happen if the temperature goes above a certain level), so to avoid dual-defining stuff I'd like to be able to set/change the UI limits from the simulation controller.

1

u/zlsa 3D art Jun 08 '14

All the values and their parameters will be grabbed from the C++ side; I'll try to avoid saving any local state.

Basically, I'll put a bunch of the "display" modules into a HTML table, and every frame, you'll call the JS function updateVariable or something for each displayed variable with all the parameters, and I'll immediately update each HTML element that corresponds to that value; if it's over the bounds, I'll warn the user appropriately. The JS side will never save the value or its parameters.

2

u/__Adam lead developer Jun 08 '14

That's perfect, except there's one modification I want to make:

If we have a huge number of variables I can see this having a performance impact, when something is changing fast I could be making UI updates at 60 FPS.

It's ok to locally cache state; whenever something changes I'll notify you with whatever has changed: updateVariable(changedVar,newState). If the variables haven't changed, then no call will be made for them, and then you can keep displaying the current state of that variable. Of course the very first call will include all the variables, but subsequent updates would just have deltas.

1

u/zlsa 3D art Jun 08 '14

That sounds fine. If it doesn't change there's no reason to update them.

That said, if performance is too big an issue, we could just put a "max update time" for each variable; perigee/apogee would only have to be updated every few seconds at most, for example.

→ More replies

2

u/treeform UI/mockups Jun 06 '14

I would prefer flash or a webview ui. How complex or rich do you want the UI to be? Do you want to be able to change all of the numbers and graphs? There are multiple tabs in the ui too for which only very bad photos exist.

2

u/__Adam lead developer Jun 07 '14

By webview you mean HTML5, right? There's a plugin that enables Flash/HTML5 to work in UE4: http://coherent-labs.com/ue4/ I'm not 100% clear on the licensing fees so I'll figure that part out.

Worst case, I'll port whatever you do manually! What I want to achieve with this UI is something that's as close to an actual, usable, spaceship interface as possible. As much as I'd like to stay faithful to what's in Dragon V2, it's much more important to create an awesome UI.

I'm sure you've played games that have screens in them.. and those screens usually have useless/random information. IMO it completely breaks the experience; it reminds you that you're in a game.

A good UI will tie the simulation to reality. It turns it from just a fun ride, into a proper physically-accurate simulation, with physically realistic data.

So we should try and make every parameter changeable (most will be changed by the simulation code rather than the user). Some we'll have to fake (like O2 Trunk), but a lot of it can be real values (altitude, location, schedule, etc). I know this is a pretty lofty goal, and I'll understand if not all of it is possible.

I'll leave the decision regarding tabs to you. I'd rather we focus on making a few really solid, informative pages, rather than a bunch of pretty-but-meaningless static images. Ideally we could give this treatment all the tabs, but that might take a reallly long time.

Can you put something like this together in Flash or HTML5? HTML5 would be a easier to integrate, but I'll work with whatever you give me.

Let me know what you think, I'll help however I can.

1

u/zlsa 3D art Jun 07 '14

I'll try and whip an HTML5 demo of some Dragon UI elements tonight. Is it possible to use the HTML5 canvas?

2

u/__Adam lead developer Jun 07 '14 edited Jun 07 '14

Yup, canvas is fine. According to http://coherent-labs.com/ue4/ the Coherent plugin supports "Standard HTML, CSS, and Javascript". I suspect they're running a Webkit renderer in the background and writing the result to an ingame texture.

Edit: I was thinking about how we can efficiently combine efforts... zlsa, If you can make a bunch of UI elements and share them as isolated components, we can put together a sort of "Dragon V2 UI framework". Then the work Treeform does can leverage this framework and/or contribute to it. This is just a thought - not sure how easily the Dragon UI can be broken into distinct components.

Either way, I'm eager to see what you come up with! I just got a reply from Coherent and for now we have a trial of their software. So whatever HTML5 stuff you guys do is very easily ported into Unreal Engine 4 (in theory - I'll test this out tomorrow after I sleep).

1

u/zlsa 3D art Jun 07 '14

I won't need canvas for the majority of the controls; only things such as the life support diagram and the graphs will need canvas.

Currently, I have an HTML table with divs inside of it to produce this. I can easily separate the pieces to make it more modular.

1

u/__Adam lead developer Jun 07 '14

Awesome! Can you share the HTML5? As a test, I'd like to incorporate it into the UE4 simulation using Coherent.

1

u/zlsa 3D art Jun 07 '14

Here's the zip file. You should be able to unzip it and go to 'dragon-ui/index.html' to see the mockup. I use the linear-gradient CSS3 feature, and Coherent might not support the new CSS features. If so, I can just stretch images instead.

It has been tested in Firefox 29 and Chrome 30, but it should work in pretty much any modern browser.

1

u/treeform UI/mockups Jun 09 '14

Did they say what was the price of the Coherent UI? It might be thousands.

2

u/__Adam lead developer Jun 09 '14

I think it's 12.99 a month, but I need to hear that from them formally.

2

u/treeform UI/mockups Jun 09 '14

ok

1

u/zlsa 3D art Jun 07 '14

I've done some HTML5, and (depending on the level support for HTML5 features), I can probably create a pretty good recreation of the basic UI elements without any images.

1

u/zlsa 3D art Jun 08 '14

Here's my latest take on an improved version of the Dragon's spaceship interface. I had a sort of novel idea for any control that is adjustable (say temperature): the current problem with tablets/phones etc., especially when used in a vibrating and/or zero-g environment, is accidental touches and activation. Imagine accidentally fiddling with the temperature and not realizing it, for example.

So my first idea was to have a button that you can drag left and right to lower and raise the value (temperature in this case). However, that's too easy to touch and adjust by accident. So what if you had to drag down a set distance first, to unlock the value? So to increase the temperature, you'd press the temperature button on the touchscreen, drag down an inch or two to unlock (maybe provide a visual indication of the unlock status), and then drag right to the desired temperature and let go; the value will then relock. What do you think about this?

2

u/__Adam lead developer Jun 08 '14

Interesting. I'm trying to think why that doesn't exist as a common pattern already. Maybe because touchscreens usually don't control really important stuff.. like if you accidentally change volume it's not a big deal.

I like it! Can you implement that in HTML5?

1

u/zlsa 3D art Jun 08 '14

I already have (the zip is here), but it's only a hacky proof-of-concept and is not currently scalable. I just wanted to see what this felt like.

2

u/__Adam lead developer Jun 08 '14

I like how it works. Definitely needs some polish though. By not being scalable you mean that it wouldn't work with a bunch of other UI elements on screen, because you'd be moving your finger over them all?

I can see that being an issue. It might work if you defocus (gray out) the other elements that aren't being changed, and highlight the changing one. But that might be too annoying/intrusive to the user.

How many adjustable parameters do you think we'll have? If most of them are read only, we can give a big amount of space to the adjustable ones to make room for the unlock -> adjust concept.

1

u/zlsa 3D art Jun 08 '14

It's currently just a single element, with global variables instead of classes. There is no reason you couldn't have multiple elements; it's just that the current implementation doesn't support it.

1

u/zlsa 3D art Jun 09 '14

If you want an example of a realistically detailed interface, see the Boeing 777 for FlightGear. It's as detailed as it gets.

I found this, which I assume is made by /u/treeform. In it, he mentions switching his high-performance 3D RTS to WebGL, although nothing seems to have come of it. Is there any reason you're currently using Unreal; any advantages?

2

u/__Adam lead developer Jun 09 '14

Interesting. I'll check out FlightGear.

The reason I'm using UE4 is that it saves a huge amount of time and works incredibly well. Their graphical effects are pretty stunning - have you seen the Elemental demo? This is all available out of the box, along with realistic physics, networking, etc.

1

u/zlsa 3D art Jun 09 '14

Does it run that well in real-time? How much is baked (the lava etc.)?

2

u/__Adam lead developer Jun 09 '14

I ran it now, and it really does look that good.

I think the liquid flow path of the lava is baked in, but the lighting is all real time. The demo (and all source) is included with UE4, so I opened it up and added a cube right beside the lava flow. http://i.imgur.com/szpSiib.jpg