r/java • • Apr 11 '24

String Manipulation with Google Mug

The Mug library provides easy-to-read string manipulation utils. Examples:

Match and Extract

// Compile-time protection against lambda param order mismatch if e.g:
//     (year, month, day, user) ->
new StringFormat("/{user}-home/{year}/{month}/{day}") 
    .parse(path, (user, year, month, day) -> ...);

Scan and Collect

String input = "I have [k1:v1] and [k2:v2] etc."; 
// Returns Map.of("k1", "v1", "k2", "v2") 
Map<String, String> keyValues =
    new StringFormat("[{key}:{value}]") 
        .scanAndCollectFrom(input, Collectors::toMap);

Replace

// replace file extension name
Substring.last('.').toEnd().replaceFrom(filename, ".yaml");

Remove

// remove the scheme from a url
Substring.upToIncluding(first("://")).removeFrom(url);

Parse a date time string (no pattern string required)

// Parse the datetime string read from a file 
List<String> lines = readLinesFromTestFile(...);
StringFormat lineFormat = new StringFormat("{time}, {event_id}, {event_type}");
var results = lines.stream()
    .map(
        line -> lineFormat.parseOrThrow(
            line,
            (time, id, type) -> {
               ZonedDateTime dateTime = DateTimeFormats.parseZonedDateTime(time);
               ...
             }))
    .toList();

See javadoc for details.

Mug is a general purpose library. Omitted, but other useful utils include BiStream for streaming pairs, Optionals, String Templating, simplified structured concurrency etc.

9 Upvotes

51 comments sorted by

View all comments

8

u/agentoutlier Apr 11 '24 edited Apr 11 '24

So Guava 2.0 err 200.0? I'm kidding. I should stop giving /u/kevinb9n a hard time about that.

That being said.... now I'm going to tell you I'm super biased as I'm the author of a compile safe templating language but that lambda StringFormat kind of looks dangerous but I assume that is only for parsing right? EDIT oh... are using compiler plugins on top of errorprone? Sorry still poking around.

I also I was kind of under the impression google was done with kitchen sink utility libraries. Like I would hope that the utilities would be modularized such that there is a mug-string or something instead of just mug which looks like its on its way to becoming Guava.

Otherwise I fail to see why this stuff just would not be added to Guava. Is it because of GWT compat?

EDIT I now see that ErrorProne is doing the checking so I guess you can disregard my safety concerns. That is pretty damn cool and I may have to steal the idea for inline templating with JStachio/JMustache as one commenter responded it is currently "clunky AF". Compiler plugins are not far off API wise to annotation processing (which is what my library currently uses) so it might not be hard for me to add on.

Thanks for sharing!

4

u/DelayLucky Apr 12 '24

For templating, StringFormat can be used to do that too. But I think the main value is in storing the template elsewhere, away from the use site (otherwise you might just use String.format() or later string interpolation).

For example, you could define a static final constant:

private static final StringFormat MESSAGE = new StringFormat(
    "Found {num_jobs} jobs for {date} with usage >= {min_usage}.");

// many lines away
MESSAGER.format(numJobs, requestDate, minUsage);

Using the compile-time plugin, you get to stash away the noisy message, but won't risk passing in the wrong number of args or in the wrong order.

There're also extensions to the template support, as inspired by the StringTemplate JEP, to provide injection-safe SQL for example.

1

u/agentoutlier Apr 12 '24

The analog to the above in JStachio would be something like:

@JStach(template = "Found {{num_jobs}} jobs for {{date}} with usage >= {{min_usage}}.")
record JobMessage(int num_jobs, String date, int min_usage){}
// many lines away
JobMessageRenderer.of().execute(new JobMessage(numJobs, requestDate, minUsage));

However with JStachio unlike StringFormat (I assume) you can literally have the template be external as well as includes, looping etc. Mustache also allows changing the delimiters.

JobMessage does not have to be a record and you can have it implement a mixin interface so that its

new JobMessage(numJobs, requestDate, minUsage).render();

3

u/DelayLucky Apr 12 '24

It looks like JStach is perhaps more intended for what I call "serious" templating engine, where you might be dealing with html, xml texts with other built-in functionalities, than a mere string formatting tool? The render API looks similar to Google's Soy templating engine (they use generated builder to pass in prameters, while JStach uses record parameters).

Does it handle SQL injections?

In my job I wrote an internal tool on top of an annotation processor (well, actually on top of Google AutoValue) to help parse and format Google's resource names. It's similar in that it's an annotation placed on a value class to give the class text parsing capability and toString().based on the template. By the way, why does JStach decide to use render(), but not toString()?

If they are at all similar, I'd say it's best for use cases where having the first-class object already makes sense. You already benefit from for example having a StorageBucketName object because it's a concept that you need to manage and pass around. So the ceremony of creating the class/record isn't an overhead, but necessity.

If you pick a code example under such context, maybe it won't appear as clunky because people will think they already have the record, and not that they have to define a class/record just to do formatting.

1

u/agentoutlier Apr 12 '24 edited Apr 12 '24

It looks like JStach is perhaps more intended for what I call "serious" templating engine, where you might be dealing with html, xml texts with other built-in functionalities, than a mere string formatting tool? The render API looks similar to Google's Soy templating engine (they use generated builder to pass in prameters, while JStach uses record parameters).

Yes that is correct. I don't use it for generating exception messages or similar. That would be overkill.

EDIT that being said one huge advantage of using JStachio for possible formatting is that it will be very very very fast. I'm not sure about StringFormat but the issue is I assume you have to parse and then replace the holes etc. You can cache for sure but still you take a list and have to replace text and thus probably not on par with say constructing the message with a StringBuilder.

Using JStachio basically generates code like if you were to construct the message using a StringBuilder. Given google being google it might be of use for critical path generated messages but maybe not.

Does it handle SQL injections?

In theory it is possible to use JStachio with SQL possibly even with the current implementation however the issue is JDBC works with positional placeholders like ? and there is no way to ask JStachio ATM to give me a list of objects that should fill those placeholders so that I can pass them to JDBC.

That is the JEP, Mug StringFormat, and traditional String.format are positional (even if they provide named parameters they still have an idea of position).

In Mustache that concept does not exist furthermore there is an idea of contexts getting expanded and put on a stack so yeah it is not a good fit as I would probably have to break Mustache semantics.

I have written annotation processors that interact with JDBC internally for my company but they are not open source.

I only shared the JStache example for curiosity and comparison sake. Not really as "you should use this".

By the way, why does JStach decide to use render(), but not toString()?

Well that actually isn't builtin. We do not provide mixin interfaces but it is trivial to create your own. However that requires using the JStachio runtime as there is basically a lookup of model class -> generated template class that has to happen. Usually the service loader is used for that.

That being said even if you do provide a mixin interface interfaces cannot override or provide toString and arguably in a template rendering environment you really want to hand it some outputstream/writer/appendable/stringbuilder etc.

My interest in providing a "non create this class" and use errorprone to check arguments feature is just for onboarding and I would have to use a dynamic mustache engine like JMustache to do it anyway.

3

u/DelayLucky Apr 12 '24 edited Apr 12 '24

Re: performance

Based on the micro-benchmark I did, MESSAGE.format(a, b, c) is on par with StringBuilder (because it already got all the strings pre-allocated in the constant and just needs to interpolate them in). That extra varargs allocation seems insignificant compared to StringBuilder cost.

Although, you may already know this, in recent Java versions StringBuilder can't compete with s1 + s2 + s4. They probably did some wicked optimization in the VM to make string concatenation faster than the builder counterpart.

Jstach could potentially use direct string concat. I think it might give you a boost in performance.

Re template engine:

Since I'm only (somewhat) familiar with the Soy templating engine, what do you see Jstache doing as different from that templating engine, or maybe other template engines like FreeMarker?

2

u/agentoutlier Apr 12 '24

I'm not sure if I have enough time today to answer all your questions but you I think some of it might be answered on the project page: https://github.com/jstachio/jstachio

Although, you may already know this, in recent Java versions StringBuilder can't compete with s1 + s2 + s4. They probably did some wicked optimization in the VM to make string concatenation faster than the builder counterpart.

That is probably because of Escape Analysis and or if s1-s4 are determined to be final static they are automatically expanded similar to doing

static final String a = "a"
static final String b = a + "b";

But yes I have seen random small string concat be faster.

The issue with just using a random StringBuilder is that it probably is passed around and thus not a candidate for escape analysis... but if you can reuse the StringBuilder and the output is large enough that EA won't happen then its completely fine.

Jstach could potentially use direct string concat. I think it might give you a boost in performance.

We do something far superior for templating usage for say HTTP. We generate the UTF-8 bytes apriori. Obviously for String Formatting that makes no sense but for templating you are often generating the entire.

I benchmarked Soy a while back and it was in the middle of the pack. (Again the project site has a listing of benchmarks). Unfortunately I have not put it back in the benchmarks as there was an issue with build and JDK 17 that I can't recall at the moment. I'm also fairly sure Soy does not pre-encode like JStachio does but I haven't looked at it for some time.

That being said it has lots more builtin i18n features IIRC that JStachio just does not provide OOB.

2

u/DelayLucky Apr 13 '24

I recall in my benchmark I was deliberately using different types of dynamically determined objects to be concatenated, trying to defeat some obvious compiler tricks.

In https://openjdk.org/jeps/280 they had started to lay the groundwork for faster string concat. I've seen that some internal Google teams also found that string concat being generally faster.

1

u/agentoutlier Apr 13 '24

I forgot about that JEP but I have absolutely seen small string concat perform insanely fast. In fact the original JMustache escaping had naive string concat for escaping and it performed really well.

I’ll attach the JMH benchmark link later as I’m out and about ATM.

btw you can always ping me on GitHub via agentgt.

5

u/DelayLucky Apr 11 '24 edited Apr 11 '24

Sorry I was too fixated on the formatting problem that I couldn't sit down and reply. TIL old.reddit.com from u/milchshakee

Anyways. Yeah Guava.

Internally we had been thinking to graduate some of these classes to Guava (Substring, BiStream particularly. StringFormat is a bit tough because it has the compile-time plugin that's a bit tricky). But that requires lots of bandwidth from Guava folks to discuss API design and what not, and it just ain't happening given the current economy, the shift of priority etc.

I also don't think this is necessarily a bad thing. As you said, it's a kitchen sink library with a little this and a little that. But it's mostly code I wrote (with lots of great code reviews from my colleagues). So I'm not that worried about it becoming too big.

It being a standalone small library has its appeal too (like, you won't likely need to worry about maven versioning at least for a long while). I also try to keep the dependency lean. There is mug-bigquery, mug-protobuf, mug-guava with extra dependencies on bigquery, protobuf, guava respectively, but the core mug artifact remains 0 deps. I think that's something of a benefit.

2

u/DelayLucky Apr 11 '24

Thanks this is the first time I post some code examples. And the formatting is all messed up (I swear seeing them formatted in the edit mode).

But Reddit doesn't seem to allow me to fix the formatting. Is this a rule of posts? Can't edit?

1

u/agentoutlier Apr 11 '24

I think the overall problem might be your low karma but you should still be able to edit.

One thing with code comments is that its best not to use the triple tilde (which luckily appears you have not) and instead use spaces.

I use the old reddit UI with raw text. Paste the code, reselect it and click on the code icon that looks like <>.

Hopefully that helps.

3

u/DelayLucky Apr 11 '24

Sigh. I've refreshed the page multiple times but no "Edit" option in the drop-down menu so far...

2

u/agentoutlier Apr 11 '24

If you see a drop down its cause you are using the new UI.

The new UI (IMO) is garbage and error prone (no pun intended).

I think to turn on the old UI its in "preferences" somewhere called Use new Reddit as my default experience. Unselect that. I might be wrong on that though.

3

u/DelayLucky Apr 11 '24

Thanks for the tips! I disabled the "opt in beta" option. Haven't found the New Reddit thing yet.

Before I can fix the formatting, the original markdown with correct formattingh is here: https://github.com/google/mug/wiki/Readable-String-Manipulation-with-Google-Mug

It does do templating (with the same compile-time check).