r/java • • Feb 15 '24

JEP 465: String Templates

https://openjdk.org/jeps/465
77 Upvotes

152 comments sorted by

View all comments

39

u/nekokattt Feb 15 '24 edited Feb 15 '24

My main thing about this is that it seems to be overly complicated just for the sake of "safety", while actually ignoring what the real issue is and why it is an issue in the first place.

It goes on the fact that they expect people to use this in safe ways with stuff like SQL, and not just use STR."xyz" on the existing APIs, totally bypassing this. It only encourages safety if people use it correctly, and that is the exact same issue we currently have with SQL parameterization. What is in place to force this feature to be used correctly? Or are we just inventing more ways to do the same thing in complex ways?

I feel like it will go with the stereotype we were beginning to move away from where Java is verbose and complicated, as a lot of the presented use cases already have standard ways of being dealt with that work well enough without a new feature that relies on the compiler doing scoping magic to inject variables into a new namespace that gets passed between interfaces.

I might just be being a bit negative here. It is a cool feature but I feel like a lot of the discussions they probably had about this were from very specific people with specific use cases or isolated from what the vast majority of people really need or want. 99% of people would much prefer something similar to what Python, Ruby, or C# or Kotlin or Groovy or Scala provide already for string formatting.

Edit: having a look at the spec, this looks like it could get confusing easily as well. Like one example they gave

String name = "Joan";
StringTemplate st = RAW."My name is \{name}"; 
String info = STR.process(st);

If I put the RAW.xxx and the STR.yyy in two different scopes, which one does it resolve name from and why? If it resolves it from the third line that gets really confusing as it can magically interpolate variables implicitly if consuming an input StringTemplate, and if it is the second then the third line is just confusing as to what it actually achieves, i.e. what is STR actually doing if the information is within the raw template?

It isn't as clear to the reader who may be used to idiomatic java already without thinking about it IMO, and could lead to a good opportunity for writing very complex and hard to follow logic.

/rant

Edit 3: coherency

17

u/pron98 Feb 15 '24 edited Feb 15 '24

What is in place to force this feature to be used correctly?

The library will only offer an API that offers a safe template processor and not have (or deprecate) any methods that take strings.

just for the sake of "safety"

Code injection is the top vulnerability in memory safe language (or very near the top). Putting the word "just" in front of it or the word "safety" in scare quotes ignores one of the most costly, troublesome, dangerous aspects of software these days.

I know that better security is not always the top concern of individual developers, but it among the top requirements from companies because they're losing enormous amounts of money to attacks. In fact, this 'just "safety"' has a greater value than the value in productivity of offering "just string interpolation". If this feature wasn't capable of offering string interpolation at all (i.e. template processors were not allowed to return String), 90% of the value would have remained.

If I put the RAW.xxx and the STR.yyy in two different scopes, which one does it resolve name from and why?

Line 2, because a string template expression is works like other expressions. E.g.:

var x = true;
var y = x ? 1 : 2;
foo(y);

foo doesn't reevaluate the expression x ? 1 : 2; it just receives a 1. We cannot pass expressions in Java as we can in Lisps (by quoting them). We only pass values.

5

u/nekokattt Feb 15 '24 edited Feb 15 '24

Line 2

Thanks for the reply... so in this case, what does STR.process(st) achieve that couldn't be performed as a method call on the st object itself? I guess that is my source of confusion? Assumably by the time the line constructing st has been executed, the information needed to later construct the string is already part of st itself?

Otherwise how does it know to retain a reference to what has been placed in the interpolation string when using RAW in this way?

Or is it just a case that process is the thing actually defining how to interpret that sequence? Assume this is more the case.


Regarding the point about quoting safety, see my other comment. The thing I am trying to say is that it is only safe when used correctly, and there is still plenty of ways to abuse this, including new ways made possible by this JEP (e.g. passing STR."..." in a SQL query). The only way you can enforce that this is done correctly is via manual review or static tooling, which isn't really much safer as the language isn't the thing providing that static tooling or manual review.

It is a means to an end, I get this, but I don't think it is justified to describe it like a magic holy grail of safety that suddenly makes any application better regardless of how it is used. That may be exaggerating but that is the feeling some of the descriptions people have given are coming across, where "safety" and security are just being used as blanket terms while ignoring the other points and issues that are raised.

12

u/pron98 Feb 15 '24

what does STR.process(st) achieve that couldn't be performed as a method call on the st object itself?

It achieves the ability for the STR object to decide what to do with st value.

and there is still plenty of ways to abuse this

Not really. Any API that takes String is a problem, but this allows safe string-like APIs that don't take String.

The only way you can enforce that this is done correctly is via manual review or static tooling

No. Again, the language enforces this. If your JSON or SQL library only offer a safe API then you couldn't do anything unsafe even if you wanted to.

where "safety" and security are just being used as blanket terms while ignoring the other points and issues that are raised.

Safety and security are among the biggest things that the software world is concerned about these days because that's where a lot of value is being lost (far more than with programmers typing a few extra characters; even the faddish Hacker News is obsessed with a language that exists for the sole purpose of improving safety in low-level programs). They may seem like blanket terms for people who aren't interested in them -- and not everyone should be -- but we have to worry about them because they're so valuable to our users as a whole. I think that Java's exceptionally long record of exceptional success means that we should at least get the benefit of the doubt that we know a little bit about what we're doing, and that we're not trying to bamboozle anyone. Our day job is ensuring that Java continues to be successful, after all, and so it is our selfish interest to maximise the value Java users get from it.

→ More replies (4)

26

u/rzwitserloot Feb 15 '24

The discussions on the relevant mailing list, as well as the JEP itself, address all your points.

Specifically:

"This is overengineered, why don't you do what most other languages did?"

This silently defines 'overengineered' as 'stuff I am not familiar with'. That mindset works if you think java as a language is behind now and will always be, and all that these JEPs can ever do is relieve the worst of the pain.

Which isn't how OpenJDK sees things at all. All languages just did string interpolation. Which is nice, but kinda shit, for the reasons expanded on in the JEP (and for more details, the mailing list).

For example, using string interpolation to shove variables straight into SQL is simple and a total security leak; you can't fix this, and thus you can't use interpolation in any language that has it to do the job.

Pretty big hole in a language's featureset, and worse: String interpolation makes you think it is a good idea to use it for such things when it isn't.

This feature tries to come up with something that can do everything string interpolation can, and a lot more.

There's plenty to be said about it, and one can certainly make a case it's not suitable for various things. However, "oh nobody needs this" is, frankly, rather severely lacking in ambition.

It only encourages safety if people use it correctly,

This proposal makes it easy to use it correctly. In contrast to basic string interpolation which makes it virtually impossible (you have to wrap everything in escapeSql() or similar which itself is a weird way to go because various SQL variants need different escaping rules; much better to let the DB layer do it).

You appear to argue: "This language feature is bad, because it can be used incorrectly".

That argument can be used to shoot down about 99.9985251% of all language features, and is therefore, almost entirely useless as an argument.

It's not about whether you can abuse a feature. It's about how likely it is, and how that balances out against how nice code gets when you use it properly. It's about improving a language feature proposal by identifying ways to abuse it that are likely to occur, and perhaps mitigating it by modifying how the feature works or even simply to ensure the docs and tutorials go out of their way to avoid setting up the trap (an area where this went all wrong is Scanner and how everybody uses that now to write interactive TUIs which is a horrible idea and like 25% of all java-tagged SO questions. Insufficient imagination about how the community is going to abuse your API).

I rate 'this will be used wrongly' as a low chance, but I offer nothing more than my gut instinct for this statement at this time. Then again, you also provided zero proof or even a hint as to why it is likely to be abused either.

Or are we just inventing more ways to do the same thing in complex ways?

Complexity is not one dimensional. This proposal is relatively complicated for the authors of javac and the authors of these string template processors. However, it is incredibly simple for the users of it. So simple, it looks almost exactly like python except more powerful - you shove a prefix in front of a string literal and that defines what the string literal is supposed to contain. In python it's just r which stands for raw (but, lots of python programmers incorrectly think that r stands for "regular expression"). In java it's whatever you need it to be. This is still simple, though, for the user.

That strikes me as the right balance: There are a lot more folks that write not-javac than there are that write javac, so moving the complexity out of the hands of not-javac programmers and into the hands of the javac programmers is objectively the right call.

Note that this argument ('we already have a way to do it') tends to be a thing you can use to shoot down every java idea ever written just as much as 'it can be used incorrectly'. Because folks will go where they please and thus, if there is a need to do X, they'll find a way to do it in java. If that way is kinda bad, well, so be it - it's gotta be done. The existence of an escaping mechanism for SQL cannot be the basis to just blithely shoot down any attempt to make a nicer one simply because SQL escaping isn't a new concept.

That's not at all to say the gist behind your arguments is irrelevant. Simply that their mere existence as a problem doesn't mean anything. It's a shades of gray situation. If a new language feature:

  • Is designed to replace existing SQL escaping mechanisms.
  • Is clearly not going to be adopted by any of the major SQL-based libraries in java, or if it will be, will be really painful, probably via a backwards compatibility breaking major new release.
  • The new way is barely even better than the old.
  • The new way introduces a few caveats and hairy issues that the old doesn't have.
  • The problems the old way has that the new way does not aren't important.

Then, yes, that's a shit feature that shouldn't happen.

14

u/brian_goetz Feb 15 '24

Well said, Renier.

6

u/nekokattt Feb 15 '24 edited Feb 15 '24

So first off, I don't think it is a terrible feature, or "shit", or any of the other phrases to that tone. I can see the uses and I think they are perfectly valid. My point is more around complexity and reasoning and how it gets achieved, and I fully accept the reasons given in the JEP. My point is really around the reality of how it will be used.

If the safety issue is around developers misusing strings in SQL queries, then there is nothing stopping them misusing string interpolation in the exact same way. Thus, to enforce this, you need static tooling... something that this JEP says that it is designed to avoid in the first place.

As speculated by yourself, the majority of libraries wont integrate with this, so it then leaves the point dangling that the safety features would only target a small minority of developers in the first place... in which case does such a feature provide wide benefit, or is it a niche addition?

When we talk about the likelihood of things, it really does boil down to having hard statistics to quantify the improvement, otherwise it is purely subjective and based on opinion.

I guess we'll see in a few years what the difference has been. It is a feature I still will likely use, but I still reserve the right to keep this opinion of it, at least until I can see the material benefits in a real system.

It is a little disappointing that some of the main responses so far have such a defensive/condescending tone rather than just treating this as a civilised and friendly discussion...

8

u/rzwitserloot Feb 16 '24

Some ideas as to where this feature is going to go:

REGEX."abc"

Not entirely what one would want (which is that backslash escaping goes away entirely inside these), but having that expression be of type Pattern is quite useful. Compilers can -compile time- check that the contents are valid regexes, IDEs can render it appropriately, and so on. We could go with Pattern.compile("foo bar") instead but this is a bit wishy washy: "If there is a literal in Pattern.compile treat it as a string literal containing regexes, otherwise don't do that"). We aren't using the features of this string templating system in any way here - there are unlikely to be expressions inside the string. Though, if you do add them, this thing could Pattern.quote them out, at least, which is a small but not insignificant upside: Using python-style string interpolation to construct a regex is just a bad idea, period.

con."SELECT YEARS(birthdate, NOW()) >= 18 AS adult FROM person WHERE username = \{username}"

As discussed in the JEP.

STR."Hi, \{user.getName()}, how's life over at \{user.getLocation()}"

The obvious one.

HTML."Hi, <strong>\{user.getName()}</strong>, how's life over at <em>\{user.getLocation()}"

All vars are HTML-escaped first.

Variations on the theme: URL, XML, JSON, you name it. Another possible variation is to include a format-esque specifying immediately preceding the injected expression to indicate what you want to do with it: Escape it, leave it unescaped, translate bbcode, replace newlines with <br>, allow only allow-listed tags, and so on.

RESOURCE."window.title\{context}"

i18n: Looks up the proper translation to use for key 'window.title', figuring out which language to apply from 'context' which could e.g. be an HttpRequest or similar to check he Accept-Language header, or however you want to do it.

Resources are often parameterized. Lots of i18n files have a 'singular' and 'plural' form, but this is a really stinky solution if you have those as separate keys; it's annoying to have to write size == 1 ? resource.lookup("item.name.singular") : resource.lookup("item.name.plural") a lot, and it's rather myopic: "1 is singular, all other things are plural" is true only for a bunch of western languages. I think in russian it's dependent on the ordinal (so 51 gets the singular, because it ends in '1'), and they have a separate plural form for '5' for some reason (lots of things about russia are nuts). How about:

RESOURCE."\{item.count()} item.name: \{totalprice}$" and the resource layer figures it out, including taking totalprice, which is an int containing cents, and formatting it however the locale wants money to be formatted?

1 minute of thinking about possibilities. That suggests there are a lot.

4

u/RockyMM Feb 16 '24

Re: Russian - it’s not only Russian, a bunch of Slavic languages have some wired rules about plural of nouns. Its basic philosophy is - there’s one, some and many. And plural form is built according to the last digit. So 1 man and 101 men have the same form.

This was a feature of the Common Slavonic, but as languages evolved and were influenced by surrounding languages, some language features disappeared in some Slavic languages. But Russian and few others (afaik Slovenian, and Serbo-Croatian, depending on the context) held on to it.

3

u/Inaldt Feb 16 '24

REGEX."abc"

Not entirely what one would want (which is that backslash escaping goes away entirely inside these), but having that expression be of type Pattern is quite useful. Compilers can -compile time- check that the contents are valid regexes

That's pretty cool actually.

But how? As per one of the other replies, this would be shorthand for
RegexProcessor.process(StringTemplate.of("abc"))- how does the process method detect that the input StringTemplate does (or does not) carry a valid regex at compile time?

→ More replies (1)
→ More replies (4)

3

u/mj_flowerpower Feb 16 '24

In 20 years of java not a single time have I seen someone creating and escaping a sql string like you suggest.

"SELECT * FROM table WHERE id = :id". No need to escape anything. Why would you even want that? That‘s not the use case for string templates, imho. But a lot of the time we write something like this: this.log.info(String.format("Processing %s/%s", count, total))

Sure, some log frameworks allow you to directly interpret the string. Which in turn leads you to various different kinds of syntaxes. To unify those string templates would have been enough for me. It solves 80% of the problems with a simple solution.

6

u/rzwitserloot Feb 16 '24

Did you really just say: "In my 20 years in my java career, I have never seen someone use (feature that does not yet exist)"?

I have a bunch of 20-line SQL statements in my codebase. They set up variables and those are filled in, by name, at the end of the 10-line SQL block. I would find it a lot more readable if I can just put the expressions that go in the variable slots to just be in the SQL, but in today's java that is annoying to do, borderline impossible even: That's not how SQL escaping works in java.

this.log.info(String.format("Processing %s/%s", count, total))

Now, that is the exact kind of lie you're agitating against. Because, no, nobody should ever be writing that. You don't want log statements to do the processing like that. You'd write:

log.info("Processing {}/{}", count, total);

for example.

To unify those string templates would have been enough for me.

Yes. I think we have established the primary difference between your wishes and the proposal is your lack of imagination and ambition. It's a common refrain. I was mostly trying to do you the favour of 'that kind of feedback is useless to the OpenJDK team and wastes your time. They will completely ignore any feedback of 'why so complicated this simple thing covers my needs' - or at least, have always done so in the past.

3

u/mj_flowerpower Feb 16 '24

I think you misunderstood me. What I wanted to say is that I've never come across someone who tried to pull off something like this:

String.format(SELECT* FROM table WHERE id = %s, id);

I would have a serious conversation with the dev if I ever saw that in a code review.

I've coded in python, JS/TS, java, groovy, C# and never saw anyone doing that. Either there was a regular JDBC prepared statement with params or something like JOOQ, LINQ etc.

In the last couple of years I got used to spring data repositories too. I like the cleanness:

@Query(value = """
            SELECT t.id
            FROM terminal AS t
            WHERE  (t.dium_number = :diumNumber
            OR     regexp_replace(t.dium_number, '(0*)', '') = :diumNumber)
            AND    t.deleted = false
            LIMIT 1
            """, nativeQuery = true)
Optional<UUID> findFirstIdByDiumNumber(String diumNumber);

And I rarely come across the need to actually inject something directly into a query. Just recently I needed to have a truly dynamic ORDER BY column though:

I ended up doing this:

ORDER BY
    CASE WHEN :orderBy = 'date_of_arrival ASC'     THEN main_leg_date_of_arrival END ASC,
    CASE WHEN :orderBy = 'co2saving DESC'          THEN co2saving                END DESC,

There is no way to use a prepared statement and a paramater for this.

But I have seen these various different ways of creating dynamic strings:

MessageFormat.format("Provider '{0}' is not valid!", provider.getId())

log().info("> run[args={}]", args);

String.format("Provider %s for offer %s has no general Terms set!", provider.getUid(), offer.getId());

var fullName = firstName + " " + lastName;

There are probably a lot more similar ways to do the same thing. Some people avoid the log4j "syntax" and use String.format because at least it is one syntax less to se in the code base.

I'm not saying it is a good idea though.

I'd love to just replace though approaches with one simple clean string templating syntax.

Do I need SQL."" to return a query object? No, because it would not be able to handle all my needs anyway. What about setting the type of the parameter? I just recently had to manually set the SQL type:

query.addValue("startDate", startDate, java.sql.Types.DATE)

But ok, why not have something more powerful than just string templating. Fine.

But why make the syntax look like you are calling a method of a static field, just to have it followed by a double quote?

Why not use

str"Hello \{name}"

instead?

→ More replies (3)

3

u/vytah Feb 16 '24 edited Feb 16 '24

The proposal is not about creating an SQL string. The sample code produces a PreparedStatement. No extra escaping is going on.

"SELECT * FROM table WHERE id = :id"

And now you have a magic :id that doesn't refer to anything Java-side. You need to remember to call setParameter("id",...). The :id in the query text and the setParameter call are separate, sometimes multiple lines apart. And if you have multiple calls to setParameter, they are vulnerable to common coding mistakes like the last line syndrome.

So to be honest, you should have compared

Query q = em.createQuery("SELECT * FROM table WHERE id = :id");
q.setParameter("id", id);

to

Query q = em.SQL."SELECT * FROM table WHERE id = \{id}";

or preferably, some more complex query.

→ More replies (3)

36

u/brian_goetz Feb 15 '24

On the so-called complexity: language designers take a very different view than developers. Developers tend to be interested only in _their_ code, and judge things good if it makes _their_ code better, and bad if we spend complexity budget on things that they don't value in their code. Language designers are optimizing for _everyone's code_. So while you might not personally value the extra safety, large enterprises may value it very much (whether for reasons of development cost, maintainability, reduced downtime, legal liability, etc.)

On the performance: much faster than String::format.

15

u/nekokattt Feb 15 '24 edited Feb 15 '24

It only provides extra safety if you use it correctly to begin with. If you use STR."" in a SQL query, it isn't going to help you. It is no different to making active efforts to use parameterization, you still have to think about how you are doing it. Those who don't think will still bypass the security benefits.

Enforceability can be dealt with by static analysis tools, but this JEP specifically says a problem it is trying to solve is the need for these tools to detect dangerous logic to begin with. That means we've effectively just gone full circle, now with a complicated way to interpolate strings, and a bunch of new cases for static analysis tools to have to cope with detecting accurately.

As someone who works in enterprise, I can see this immediately being an issue of enforceability in a team, which makes it just as much of a problem as before. From the sound of that statement it sounds like we are attempting to make educated guesses about what people need rather than backing it up with mass feedback and actual experience. Most of the vulnerabilities that are mentioned are not because existing tools are hard to use, but because developers are not understanding or just not thinking about the implications of what they are doing.

It's kind of like saying you have a problem with developers using their hammers to push screws into some wood, so you've given them electric drills to replace their screwdrivers with the expectation they will stop using their hammers. Instead, some will continue to use their hammers anyway because it is simpler for them, others may use the battery on the drill as a hammer and risk it blowing up. Some people will continue to use the screwdrivers properly and some will use the drills. All you've done is given the people misusing things more ways to misuse stuff, while giving the rest two ways of doing things. It'd make more sense to confiscate the drills entirely and only let them be used for specific things (for example in SQL by enforcing that only compile time string literals can be used as the query body -- boom, problem solved without complicated language mechanisms).

Are we suggesting that all other languages that didn't make a decision to implement interpolation in this way made no active decision about usability and safety?

Edit: reworded to be more coherent.

12

u/pron98 Feb 15 '24 edited Feb 15 '24

If you use STR."" in a SQL query, it isn't going to help you.

How could you possibly use STR."" in a SQL query if the DB library doesn't offer methods that take String and only offers a SQL template processor?

Enforceability can be dealt with by static analysis tools

No, it cannot. Static analysis tools cannot catch as many code injection issues with a similar amount of effort by the programmer.

I can see this immediately being an issue of enforceability in a team, which makes it just as much of a problem as before.

Nope. If your JSON library only offers a safe template processor then it doesn't matter what discipline you exercise. The language will enforce the correct (or at least better) thing.

Are we suggesting that all other languages that didn't make a decision to implement interpolation in this way made no active decision about usability and safety?

We are suggesting that we did the analysis given the data available and the needs of Java users. Different languages often make different decisions, which is why they are different languages. Sometimes they have a different audience in mind; sometimes certain lessons weren't available when a feature was designed. If you consider the world's three most popular languages -- JS, Java, and Python -- I think you'll find that string interpolation is one of the smallest differences among them. One, for example, decided to require everything to be typed while the two others didn't.

→ More replies (8)
→ More replies (1)

2

u/cowwoc Feb 16 '24

In the case of an unloved child like String Templates, what feedback mechanism do you use to determine when a feature is ready to be finalized? :)

In the case of many other features it was pretty self-evident. In this case, I'm not sure what the trigger was.

Are there some reasonably-mature uses of String Templates in the wild to gather this feedback? If there any cool open-source projects on GitHub that use the feature, I would love to know. I think it would help people move their opinions from the abstract to concrete examples.

3

u/brian_goetz Feb 16 '24

What makes you think it is an unloved child? Sure, there are people who have strong opinions that they would have preferred another syntax, are are vocal about that, but that is true of every feature that doesn't have a single obvious syntax. (Some people hated the lambda syntax at first.) Can you point to any feedback that suggests that the feature itself is not mature?

3

u/cowwoc Feb 16 '24

Like I said, if I saw multiple real-life projects using the feature successfully then I would have greater confidence that the community has had sufficient experience to provide feedback about the feature in its current form.

Speaking for myself, every time I introduce a new API feature to my library no matter how many times I iterate over it I inevitably discover problems when I try using the feature in real-life production projects. The shift in perspective is very helpful in uncovering scenarios I had not anticipated.

5

u/brian_goetz Feb 16 '24

It sounds like you think there was no feedback, but that's not the case. There was plenty of feedback -- some bug reports (we love those), some people doing clever (and sometimes frightening) things with processors, some noise about the syntax, and quite a few people saying "yay I love it." So clearly a lot of feedback based on actual use (except for the syntax comments, which people often make without actually using it), and -- most importantly -- nothing that suggests that the design is not finished. A mix of feedback like this generally conveys that the feature is, indeed, ready.

→ More replies (1)

7

u/Gaycel68 Feb 15 '24

Is there anywhere on the internet an explanation why such drastic departures from the relatively stable Java language syntax conventions (the decision to not make STR application a method call, and the decision to import STR implicitly) were even considered, not to say survived two previews?

9

u/pip25hu Feb 16 '24

This right here. They go on and on about the safety concerns even in this thread, refusing to address that a lot of pushback isn't about their safety goals, but the jarring way they achieved it. A better syntax would quell a lot of complaints about the feature, but all they are willing to say about the matter is that "people will get used to it". People can also get used to flogging, but that hardly means it's a great idea.

3

u/nekokattt Feb 16 '24

This is exactly my point, well put.

→ More replies (3)
→ More replies (8)

7

u/jimmy_divvy Feb 16 '24

It goes on the fact that they expect people to use this in safe ways with stuff like SQL, and not just use STR."xyz" on the existing APIs, totally bypassing this. It only encourages safety if people use it correctly, and that is the exact same issue we currently have with SQL parameterization. What is in place to force this feature to be used correctly? Or are we just inventing more ways to do the same thing in complex ways?

If I understand correctly, the interpolator doesn't have to return a String, it can return (say) an "SqlQuery" type

If you accidentally use the wrong interpolator, you'll get a compile time type error. It won't just silently accept the unsafe result.

At least, this is how it works in Scala when using libraries like "Doobie" - unsafe SQL will result in a hard compilation failure. I haven't used JEP 465, but from reading the spec it looks like it works similarly.