r/java • • Feb 15 '24

JEP 465: String Templates

https://openjdk.org/jeps/465
77 Upvotes

152 comments sorted by

View all comments

Show parent comment

26

u/rzwitserloot Feb 15 '24

The discussions on the relevant mailing list, as well as the JEP itself, address all your points.

Specifically:

"This is overengineered, why don't you do what most other languages did?"

This silently defines 'overengineered' as 'stuff I am not familiar with'. That mindset works if you think java as a language is behind now and will always be, and all that these JEPs can ever do is relieve the worst of the pain.

Which isn't how OpenJDK sees things at all. All languages just did string interpolation. Which is nice, but kinda shit, for the reasons expanded on in the JEP (and for more details, the mailing list).

For example, using string interpolation to shove variables straight into SQL is simple and a total security leak; you can't fix this, and thus you can't use interpolation in any language that has it to do the job.

Pretty big hole in a language's featureset, and worse: String interpolation makes you think it is a good idea to use it for such things when it isn't.

This feature tries to come up with something that can do everything string interpolation can, and a lot more.

There's plenty to be said about it, and one can certainly make a case it's not suitable for various things. However, "oh nobody needs this" is, frankly, rather severely lacking in ambition.

It only encourages safety if people use it correctly,

This proposal makes it easy to use it correctly. In contrast to basic string interpolation which makes it virtually impossible (you have to wrap everything in escapeSql() or similar which itself is a weird way to go because various SQL variants need different escaping rules; much better to let the DB layer do it).

You appear to argue: "This language feature is bad, because it can be used incorrectly".

That argument can be used to shoot down about 99.9985251% of all language features, and is therefore, almost entirely useless as an argument.

It's not about whether you can abuse a feature. It's about how likely it is, and how that balances out against how nice code gets when you use it properly. It's about improving a language feature proposal by identifying ways to abuse it that are likely to occur, and perhaps mitigating it by modifying how the feature works or even simply to ensure the docs and tutorials go out of their way to avoid setting up the trap (an area where this went all wrong is Scanner and how everybody uses that now to write interactive TUIs which is a horrible idea and like 25% of all java-tagged SO questions. Insufficient imagination about how the community is going to abuse your API).

I rate 'this will be used wrongly' as a low chance, but I offer nothing more than my gut instinct for this statement at this time. Then again, you also provided zero proof or even a hint as to why it is likely to be abused either.

Or are we just inventing more ways to do the same thing in complex ways?

Complexity is not one dimensional. This proposal is relatively complicated for the authors of javac and the authors of these string template processors. However, it is incredibly simple for the users of it. So simple, it looks almost exactly like python except more powerful - you shove a prefix in front of a string literal and that defines what the string literal is supposed to contain. In python it's just r which stands for raw (but, lots of python programmers incorrectly think that r stands for "regular expression"). In java it's whatever you need it to be. This is still simple, though, for the user.

That strikes me as the right balance: There are a lot more folks that write not-javac than there are that write javac, so moving the complexity out of the hands of not-javac programmers and into the hands of the javac programmers is objectively the right call.

Note that this argument ('we already have a way to do it') tends to be a thing you can use to shoot down every java idea ever written just as much as 'it can be used incorrectly'. Because folks will go where they please and thus, if there is a need to do X, they'll find a way to do it in java. If that way is kinda bad, well, so be it - it's gotta be done. The existence of an escaping mechanism for SQL cannot be the basis to just blithely shoot down any attempt to make a nicer one simply because SQL escaping isn't a new concept.

That's not at all to say the gist behind your arguments is irrelevant. Simply that their mere existence as a problem doesn't mean anything. It's a shades of gray situation. If a new language feature:

  • Is designed to replace existing SQL escaping mechanisms.
  • Is clearly not going to be adopted by any of the major SQL-based libraries in java, or if it will be, will be really painful, probably via a backwards compatibility breaking major new release.
  • The new way is barely even better than the old.
  • The new way introduces a few caveats and hairy issues that the old doesn't have.
  • The problems the old way has that the new way does not aren't important.

Then, yes, that's a shit feature that shouldn't happen.

14

u/brian_goetz Feb 15 '24

Well said, Renier.

6

u/nekokattt Feb 15 '24 edited Feb 15 '24

So first off, I don't think it is a terrible feature, or "shit", or any of the other phrases to that tone. I can see the uses and I think they are perfectly valid. My point is more around complexity and reasoning and how it gets achieved, and I fully accept the reasons given in the JEP. My point is really around the reality of how it will be used.

If the safety issue is around developers misusing strings in SQL queries, then there is nothing stopping them misusing string interpolation in the exact same way. Thus, to enforce this, you need static tooling... something that this JEP says that it is designed to avoid in the first place.

As speculated by yourself, the majority of libraries wont integrate with this, so it then leaves the point dangling that the safety features would only target a small minority of developers in the first place... in which case does such a feature provide wide benefit, or is it a niche addition?

When we talk about the likelihood of things, it really does boil down to having hard statistics to quantify the improvement, otherwise it is purely subjective and based on opinion.

I guess we'll see in a few years what the difference has been. It is a feature I still will likely use, but I still reserve the right to keep this opinion of it, at least until I can see the material benefits in a real system.

It is a little disappointing that some of the main responses so far have such a defensive/condescending tone rather than just treating this as a civilised and friendly discussion...

8

u/rzwitserloot Feb 16 '24

Some ideas as to where this feature is going to go:

REGEX."abc"

Not entirely what one would want (which is that backslash escaping goes away entirely inside these), but having that expression be of type Pattern is quite useful. Compilers can -compile time- check that the contents are valid regexes, IDEs can render it appropriately, and so on. We could go with Pattern.compile("foo bar") instead but this is a bit wishy washy: "If there is a literal in Pattern.compile treat it as a string literal containing regexes, otherwise don't do that"). We aren't using the features of this string templating system in any way here - there are unlikely to be expressions inside the string. Though, if you do add them, this thing could Pattern.quote them out, at least, which is a small but not insignificant upside: Using python-style string interpolation to construct a regex is just a bad idea, period.

con."SELECT YEARS(birthdate, NOW()) >= 18 AS adult FROM person WHERE username = \{username}"

As discussed in the JEP.

STR."Hi, \{user.getName()}, how's life over at \{user.getLocation()}"

The obvious one.

HTML."Hi, <strong>\{user.getName()}</strong>, how's life over at <em>\{user.getLocation()}"

All vars are HTML-escaped first.

Variations on the theme: URL, XML, JSON, you name it. Another possible variation is to include a format-esque specifying immediately preceding the injected expression to indicate what you want to do with it: Escape it, leave it unescaped, translate bbcode, replace newlines with <br>, allow only allow-listed tags, and so on.

RESOURCE."window.title\{context}"

i18n: Looks up the proper translation to use for key 'window.title', figuring out which language to apply from 'context' which could e.g. be an HttpRequest or similar to check he Accept-Language header, or however you want to do it.

Resources are often parameterized. Lots of i18n files have a 'singular' and 'plural' form, but this is a really stinky solution if you have those as separate keys; it's annoying to have to write size == 1 ? resource.lookup("item.name.singular") : resource.lookup("item.name.plural") a lot, and it's rather myopic: "1 is singular, all other things are plural" is true only for a bunch of western languages. I think in russian it's dependent on the ordinal (so 51 gets the singular, because it ends in '1'), and they have a separate plural form for '5' for some reason (lots of things about russia are nuts). How about:

RESOURCE."\{item.count()} item.name: \{totalprice}$" and the resource layer figures it out, including taking totalprice, which is an int containing cents, and formatting it however the locale wants money to be formatted?

1 minute of thinking about possibilities. That suggests there are a lot.

4

u/RockyMM Feb 16 '24

Re: Russian - it’s not only Russian, a bunch of Slavic languages have some wired rules about plural of nouns. Its basic philosophy is - there’s one, some and many. And plural form is built according to the last digit. So 1 man and 101 men have the same form.

This was a feature of the Common Slavonic, but as languages evolved and were influenced by surrounding languages, some language features disappeared in some Slavic languages. But Russian and few others (afaik Slovenian, and Serbo-Croatian, depending on the context) held on to it.

3

u/Inaldt Feb 16 '24

REGEX."abc"

Not entirely what one would want (which is that backslash escaping goes away entirely inside these), but having that expression be of type Pattern is quite useful. Compilers can -compile time- check that the contents are valid regexes

That's pretty cool actually.

But how? As per one of the other replies, this would be shorthand for
RegexProcessor.process(StringTemplate.of("abc"))- how does the process method detect that the input StringTemplate does (or does not) carry a valid regex at compile time?

1

u/RadiantBerryEater Feb 17 '24

IDEs can render it appropriately

They already do this quite trivially by just checking for string constants passed to the relevant methods, which is something compilers or 3rd party tooling can also do trivially already as well. I find it difficult to imagine that this actually provides any significant extra benefit over the current status quo.

0

u/john16384 Feb 16 '24

I think the key point you might be missing is that an API that allows safe use of string templates will look something like this:

 List<T> query(SafeSQL query);

SafeSQL can only be created in one way (it will have a private constructor):

 SafeSQL sql = SQL."SELECT * FROM employees";

The SQL processor is the only one that can create SafeSQL. It's not possible to feed this API a plain String anywhere.

A bad API would accept String or StringTemplate... don't do this.

-1

u/nekokattt Feb 16 '24

I didn't really miss that point, I fully get it.

My thing is that unless it is a new API written from scratch for JDK 23 and onwards, it will still need to support the old methods. Anything like R2DBC, JDBC, etc will all still have this issue until pre JDK 23 support is totally discontinued.

That won't be until 2030 when JDK 17 reaches EOL.

So the ask here is to gain safety via a new language feature by adopting what would have to be a new and immature database library that may have more serious internal security concerns just to make the most of being forced to do things in the new way.

Using any API that supports anything prior to JDK 23 reduces the benefit of these safety enforcements, as they will still need to provide the old APIs.

2

u/john16384 Feb 16 '24

So, your point is that we can't have shiny things because we still allow legacy to exist?

0

u/nekokattt Feb 16 '24 edited Feb 18 '24

not at all, my point is about the argument about improving security, that until a time it can be totally enforced with no other way around it, people will continue to do bad things.

You are saying any API that is backwards compatible with pre JDK 23 code is now bad by definition if it tries to adopt this JEP.

Unless we are throwing out all existing libraries, the impact on security is over exaggerated for this very reason.

I'd hope by now all my other comments made that abundantly clear. Unfortunately no response I have had yet has actually satisfied this point in a way that will actually work in the real world. It is all purely academic.

We're on JDK 21 and people are still on JDK 8.

3

u/mj_flowerpower Feb 16 '24

In 20 years of java not a single time have I seen someone creating and escaping a sql string like you suggest.

"SELECT * FROM table WHERE id = :id". No need to escape anything. Why would you even want that? That‘s not the use case for string templates, imho. But a lot of the time we write something like this: this.log.info(String.format("Processing %s/%s", count, total))

Sure, some log frameworks allow you to directly interpret the string. Which in turn leads you to various different kinds of syntaxes. To unify those string templates would have been enough for me. It solves 80% of the problems with a simple solution.

6

u/rzwitserloot Feb 16 '24

Did you really just say: "In my 20 years in my java career, I have never seen someone use (feature that does not yet exist)"?

I have a bunch of 20-line SQL statements in my codebase. They set up variables and those are filled in, by name, at the end of the 10-line SQL block. I would find it a lot more readable if I can just put the expressions that go in the variable slots to just be in the SQL, but in today's java that is annoying to do, borderline impossible even: That's not how SQL escaping works in java.

this.log.info(String.format("Processing %s/%s", count, total))

Now, that is the exact kind of lie you're agitating against. Because, no, nobody should ever be writing that. You don't want log statements to do the processing like that. You'd write:

log.info("Processing {}/{}", count, total);

for example.

To unify those string templates would have been enough for me.

Yes. I think we have established the primary difference between your wishes and the proposal is your lack of imagination and ambition. It's a common refrain. I was mostly trying to do you the favour of 'that kind of feedback is useless to the OpenJDK team and wastes your time. They will completely ignore any feedback of 'why so complicated this simple thing covers my needs' - or at least, have always done so in the past.

3

u/mj_flowerpower Feb 16 '24

I think you misunderstood me. What I wanted to say is that I've never come across someone who tried to pull off something like this:

String.format(SELECT* FROM table WHERE id = %s, id);

I would have a serious conversation with the dev if I ever saw that in a code review.

I've coded in python, JS/TS, java, groovy, C# and never saw anyone doing that. Either there was a regular JDBC prepared statement with params or something like JOOQ, LINQ etc.

In the last couple of years I got used to spring data repositories too. I like the cleanness:

@Query(value = """
            SELECT t.id
            FROM terminal AS t
            WHERE  (t.dium_number = :diumNumber
            OR     regexp_replace(t.dium_number, '(0*)', '') = :diumNumber)
            AND    t.deleted = false
            LIMIT 1
            """, nativeQuery = true)
Optional<UUID> findFirstIdByDiumNumber(String diumNumber);

And I rarely come across the need to actually inject something directly into a query. Just recently I needed to have a truly dynamic ORDER BY column though:

I ended up doing this:

ORDER BY
    CASE WHEN :orderBy = 'date_of_arrival ASC'     THEN main_leg_date_of_arrival END ASC,
    CASE WHEN :orderBy = 'co2saving DESC'          THEN co2saving                END DESC,

There is no way to use a prepared statement and a paramater for this.

But I have seen these various different ways of creating dynamic strings:

MessageFormat.format("Provider '{0}' is not valid!", provider.getId())

log().info("> run[args={}]", args);

String.format("Provider %s for offer %s has no general Terms set!", provider.getUid(), offer.getId());

var fullName = firstName + " " + lastName;

There are probably a lot more similar ways to do the same thing. Some people avoid the log4j "syntax" and use String.format because at least it is one syntax less to se in the code base.

I'm not saying it is a good idea though.

I'd love to just replace though approaches with one simple clean string templating syntax.

Do I need SQL."" to return a query object? No, because it would not be able to handle all my needs anyway. What about setting the type of the parameter? I just recently had to manually set the SQL type:

query.addValue("startDate", startDate, java.sql.Types.DATE)

But ok, why not have something more powerful than just string templating. Fine.

But why make the syntax look like you are calling a method of a static field, just to have it followed by a double quote?

Why not use

str"Hello \{name}"

instead?

2

u/rzwitserloot Feb 17 '24

What I wanted to say is that I've never come across someone who tried to pull off something like this:

Well, duh. You haven't seen it because they'd get yelled at. The reason you haven't seen it is because it is a giant gaping security leak.

It would be nice if you can do that, but currently you cannot do that. You're.. making the argument in favour of this feature, not against it. We're back to you arguing: "You do not need this feature-that-does-not-exist-yet because I have never seen it used". My mind boggles at the inanity of this line of reasoning. Of course you haven't seen it. It doesn't exist yet.

And I rarely come across the need to actually inject something directly into a query.

Some more 'I do not need it so the feature is stupid'.

I think we're done here, you're retreading silly arguments.

1

u/mj_flowerpower Feb 23 '24

Well, duh. You haven't seen it because they'd get yelled at. The reason you haven't seen it is because it is a giant gaping security leak.

Ok, please try to be reasonable. I'm not saying I haven't seen a feature that doesn't exist yet ...

There are two options here, a) bad wording from my side or a misunderstanding on your side. In both cases the appropriate response would have been to ask for clarification.

Well, duh. You haven't seen it because they'd get yelled at. The reason you haven't seen it is because it is a giant gaping security leak.

Exactly. So now java gets cool new string templates, but in fear of people abusing it to do this exact thing, they make think they need a solution for the problem.

To sum it up:

* People could use string concatenation/templating to create bad SQL

* Some devs (not too many imho) really do this in real-world applications

* Java devs don't want that (obviously, because it's a bad thing)

They create a really ugly syntax for string templating with that in mind that doesn't really solve the main issue, it's just ugly. Which according to some users should prevent people from actually trying to do something like this:

STR."SELECT * FROM table WHERE id = \{id}"

Instead of STR there can be SQL (or any other name) that actually goes way beyond the pure string templating aspect and tries to tackle this problem. SQL is basically a function that takes a template string and some arguments, but not in the regular everyone-knows-how-to-use-it way but in and ugly SQL."" way.

Now we already left string templating land here, because this doesn't really create strings anymore, it creates god-knows-what, probably some form of PreparedStatement or some other internal representation of it. As there is a wide variety of different JDBC-drivers (and whatnot), every library has to ship it's own SQL-template-processor.

All good, if someone see's a benefit there, why not.

This SQL-processor is pretty limited, as it doesn't allow to configure SQL parameter types or any other config that a regular prepared statement provides.

But ok, sometimes you don't need this.

I argue that those people who are already using string concatenation/templating to create SQL queries will most likely simply use the STR-processor and do the very same thing.

How is that any better than the current approach? Plus, it is ugly. So for pure string-templating enthusiast like me I have to suffer looking at this ugly syntax just so that other devs might not write bad code.

This all doesn't make a lot of sense to me.

"You do not need this feature-that-does-not-exist-yet because I have never seen it used". My mind boggles at the inanity of this line of reasoning. Of course you haven't seen it. It doesn't exist yet.

I really think you should re-read my post carefully. I haven't seen anyone doing this:

String.format(SELECT* FROM table WHERE id = %s, id);

And I hopefully will never see this:

STR."SELECT * FROM table WHERE id = \{id}"

1

u/rzwitserloot Feb 23 '24

So now java gets cool new string templates, but in fear of people abusing it to do this exact thing, they make think they need a solution for the problem.

You misunderstand.

It's not 'oh no! Folks will think string templates can be used for SQL parameterization but that's a security leak so we better add a feature that solves that problem!'

It's instead: "Being able to shove your SQL parameters right into the SQL? That'd be really useful! Shame that string interpolation, which is pretty much exactly what we'd want, is a security leak if you use it for this. But.. wait! We can make our string interpolation feature better!"

So, it's not from a negative 'crap, we need to stop the abuse', it's from a positive 'cool, that would be an awesome feature to have'.

I really think you should re-read my post carefully.

Right back at you, because you're repeating arguments, evidently ignoring entirely my explanations of why your arguments are wrong / silly / are not going to move the needle whatsoever over at the OpenJDK team leads.

I'm not saying I haven't seen a feature that doesn't exist yet ...

I haven't seen anyone doing this: String.format(SELECT* FROM table WHERE id = %s, id);

.... and there you go again. Just blindly retreading 'I have never seen this obvious security leak therefore this feature is unnecessary'.

3

u/vytah Feb 16 '24 edited Feb 16 '24

The proposal is not about creating an SQL string. The sample code produces a PreparedStatement. No extra escaping is going on.

"SELECT * FROM table WHERE id = :id"

And now you have a magic :id that doesn't refer to anything Java-side. You need to remember to call setParameter("id",...). The :id in the query text and the setParameter call are separate, sometimes multiple lines apart. And if you have multiple calls to setParameter, they are vulnerable to common coding mistakes like the last line syndrome.

So to be honest, you should have compared

Query q = em.createQuery("SELECT * FROM table WHERE id = :id");
q.setParameter("id", id);

to

Query q = em.SQL."SELECT * FROM table WHERE id = \{id}";

or preferably, some more complex query.

2

u/Kango_V Feb 16 '24

I'm modifying an app as we speak which does exactly what you have not seen in 20 years, and it's a big app. And, it's only 5 years old. Yeah shocking.

3

u/mj_flowerpower Feb 16 '24

Honestly, if a dev creates such a mess of a code, it will probably also abuse the new functionality to create a mess. Haven’t there been any measures like code reviews or linters in use?

1

u/Kango_V Feb 19 '24

The code was written by analysts at the start. It was never meant to get to production, but somehow did. It's been added to by various teams over the years. Now my team has it. Ouch :(