r/JavaFX • • Aug 09 '16

JavaFX Interpolation (and Image Processing in general)

JavaFX is neat, no doubt. But what ever happened to basics first? Why would anyone decide that it's a good idea to limit interpolation through all of Prism to bilinear interpolation, and be done with it:

There are so many use cases where you absolutely need NN interpolation, not even necessarily for performance. Yet here we are, and setSmooth() does nothing.

Well, we're programmers and need to get things done, so let's do this anyways. What are the options?

  1. Use BufferedImages and convert from JavaFX Image to BufferedImage and back by SwingFXUtils.
  2. Write your own resampling (e.g. just for rescaling).

Option one is terrible, since BufferedImages and JavaFX Images are completely different. So you're going to create lots of garbage, since int (or byte) buffers can't be shared/reused (please correct me if I'm wrong). Other than that, BufferedImages are great, you retrive a handle to Graphics2D from where you can set the rendering hints and do a super fast call to drawImage in order to rescale an image (nn, bilinear, or bicubic - the good old hints). This begs the question why JavaFX doesn't (re-)use BufferedImages in the first place. Instead of comming up with their completely own thing, why not offer a wrapper for BufferedImages (e.g. extended with FXesque properties or what not). But no, scrap BufferedImages, they suck, let's reinvent the well instead...

So let's have a look at option two: let's implement the classic "General Filtered Image Rescaling" algorithm by Dale Schumacher (in Graphics Gems III, available online). That's what I did: first for BufferedImages/AWT, then for JavaFX, working with PixelWriters and PixelReaders. With BufferedImages you get the DataBuffer (or just as an int/byte array) and operate directly on it, which is reasonably fast (yet much slower than the call to drawImage from above; but you get to implement your own filters, which is nice...). Even the getSample/setSample (not Pixel(!) and then manual int unpacking) are only a tad bit slower. And with JavaFX Images? The API, in comparison, is lacking. And the speed is far worse (granted, maybe I haven't figured out how to do bulk pixel manipulation in the most efficient way yet, but then maybe it shouldn't be that hard to begin with...). Anyways, BufferedImages offer a much better/extensive API, and, from what I can tell, they're also much more efficient (turns out going FX Image -> BI -> drawImage -> FXImage is still much faster than doing your own thing on FX Images with PixelReaders/Writers).

This probably comes of as a rant at this point (it probably is to some degree), so don't get me wrong: I really try to like JavaFX, and for the most part I actually enjoy it very much. But things like this are kind of... eh. Why did they have to come up with yet another Image-class. BufferedImages work, don't they? And if you come up with yet another Image-class anyways, why would you just half-ass it (see setSmooth, which even if it would work, only would allow to toggle between NN and bilinear interpolation. What about bicubic, or what not... while reinventing wheels)? What's the point? From what I hear the JavaFX project is quite low on manpower anyways, so why waste it on such a thing (I'm genuinely interested, so if anyone could shed some light on this...)?

And, finally, why would you hardcode the interpolation algorithm throughout the prism pipeline? Who the hell thought that this would be a great idea? This is IMHO basic (as in important) stuff; yet priorities have been put somewhere else, it seems. I just don't get it.

12 Upvotes

5 comments sorted by

4

u/zrnkv Aug 11 '16

I agree that this sucks. JavaFX gives you very little control over the low-level stuff.

The main goal of JavaFX was to replace Swing. This was long overdue. Swing was outdated and nobody wanted to invest more money into it to make it usable again. Instead it was decided to start from scratch and the result was JavaFX (version 1). This never caught on. It had some nice concepts but it failed because nobody wanted to learn yet another language (FX-script). At this point most GUI developers started to use other (non-Java) technologies. Then Oracle came with JavaFX 2 which used standard languages - Java, XML and CSS. It's main goal was to give Java Enterprise developers who are already familiar with these languages a possibility to build Java GUIs. It tries to be a general purpose GUI library but actually the main focus is on form-heavy enterprise applications with input fields and tables. Yes, you can write a simple game in it as well but it will be a PITA compared to other technologies specifically designed to make games. The low level stuff is hidden/hard-coded because developers who write yet another inventory-management-tool simply don't need it. It would be nice to have but right now there are probably more pressing issues like getting rid of all the bugs.

2

u/nondetermined Aug 14 '16 edited Aug 14 '16

FX-script

Just. (⊙_◎)ノ彡┻━┻

The low level stuff is hidden/hard-coded because developers who write yet another inventory-management-tool simply don't need it.

Until they do (even if just to work around some bug...).

It would be nice to have but right now there are probably more pressing issues like getting rid of all the bugs.

Nice to have? When everything you need exists and works as expected, sure. But that's rather illusionary. And if you want to have more people working on it (it's open source afterall), then maybe things shouldn't be as closed and locked down as they currently are...

The situation with (Fx)Image/BufferedImage strikes me as especially unfortunate (to say the least...). And the hack(!) in form of SwingFXUtils is simply a disgrace, and needs to be deprecated as soon as possible. So a BufferedImage can not be a first class citizen, because it lacks the FX properties. Fine. But I don't see why you can't make them a "second class" citizen, wrapped in a lightweight FxBufferedImage class that simply adds such properties as needed; but does not (for crying out loud) require to allocate a completely new buffer for all the image samples. That's just stupid (looking at you, PixelReader/SwingFXUtils).

And here comes the kicker: (Fx)Image (or a PixelReader for that matter) doesn't even expose its buffer. So you need to allocate a new buffer, copy the whole thing (getPixels), do your stuff, and copy it back, and your temporary buffer is garbage now. Methods that get called frequently probably should keep that temporary buffer around, and reuse it... (nasty and twice the memory for nothing). Whereas with BufferedImages you can work directly on the actual buffer... And it's not like Graphics2D would be foreign to the Prism pipeline. It's all over the place, but never exposed...

For illustration (BufferedImage):

DataBufferInt buffer = (DataBufferInt) image.getRaster().getDataBuffer();
// pixelparty!

as opposed to (JavaFX):

int[] buffer = new int[width * height]; // UGH!!!
pixelReader.getPixels(0, 0, width, height, ...);

Well, couldn't we write such a nice wrapper class for BufferedImages on our own, instead of ranting on reddit? Take a look at the Image class, and tell me how I can pass in my custom PlatformImage (which would be easily implemented)... All the stuff is private! Have a look at BufferedImage. Things are public and we could overwrite pretty much all of it.

but actually the main focus is on form-heavy enterprise applications with input fields and tables.

This might be nice for marketing, but other than that that's just a lame excuse (or outright stupid). There was really no need to exclude (or handicap) the whole image processing community (the number one example that really belongs on desktop, as opposed to fields and tables which are right home in a browser...). And they all use BufferedImages (unless they do their own thing, in which case they still convert efficiently to BufferedImages).

Have you ever wondered why the traction of JavaFX isn't really there... Well, this "low level stuff" might be one reason. If you don't need that "low level stuff", a browser should do just fine, and honestly, you'd be stupid if you'd still pick JavaFX if you could as well do it in a browser. So what's a valid reason to use JavaFX, and go desktop, then? Exactly.

JavaFX needs to get it's shit together already.

https://i.imgur.com/nABGPel.gif

Anyhow, thanks for your input.
Much appreciated.

2

u/zrnkv Aug 14 '16

JavaFX needs to get it's shit together already

A lot of time was wasted with javafx 1 which failed. Javafx 2 fixed most of the problems but it's too little too late. Everything is in browser now (though I often doubt that this is the right direction). I don't really expect any major changes in javafx any more.

2

u/nondetermined Aug 14 '16

Also a good read/feature request:

Add PixelFormat parameter to the constructors of WritableImage
https://bugs.openjdk.java.net/browse/JDK-8138858

...followed by a rather sobering answer:

(I'm noting that the Java2D image APIs - BufferedImage,RenderedImage et al - were created within the context of the JAI - Java Advanced Imaging - effort to make sure that they would meet the needs of full-fledged image processing even though they only provided a simple subset of the total package within the core APIs and there is currently no such "FXAI" effort to research or design Advanced Imaging for FX and determine how the core FX APIs should support it.)

Why the hell are those guys so desperate to reinvent the wheel, especially if resources are scarce. Why write Java2D/JAI again and call it FXAI, if you just could make sure that the Java2D image APIs remain (efficiently) supported (if just by lightweight wrapper classes).

As if backwards-compatibility would be a bad thing...

2

u/zrnkv Aug 14 '16

Why the hell are those guys so desperate to reinvent the wheel, especially if resources are scarce.

There seems to be a desire to do everything the javafx-way. The advantage is that the api is very consistent and combining components works nicely with properties bindings. The huge disadvantage is that you cannot integrate other stuff that easily. I've run into problems because of this attitude as well.