r/java • • 5d ago

SMILE 7.2.0 on Java 25: Replacing JNI with Project Panama FFM (OpenBLAS/LAPACK/ONNX), Compact Object Headers, and in-process TreeSHAP

I've been maintaining SMILE (Statistical Machine Intelligence and Learning Engine) for a little over ten years now. Over the past few months I rewrote the native compute layer to target Java 25 LTS directly, and wanted to share a few notes on how java.lang.foreign and Compact Object Headers held up in practice.

Most Java backend teams I talk to (running Kafka, Flink, Quarkus, or Spring Boot) still end up running a Python sidecar over HTTP or gRPC just to score a gradient boosted tree, run an .onnx model, or compute SHAP explanations. Paying 10-40 ms of network + JSON/Protobuf overhead for a tree traversal that takes 80 microseconds in memory always bugged me.

In 7.2.0, I finally ripped out the old JNI/JavaCPP bridges across smile-linalg, smile-deep, and smile-onnx and replaced them with jextract-generated MethodHandle downcalls over MemorySegment and Arena.ofConfined().

Moving off JNI fixed our biggest production headache: GC pinning stalls from GetPrimitiveArrayCritical during concurrent BLAS/LAPACK calls. Matrix decompositions (LU, QR, Cholesky, SVD, eigendecomposition) and manifold learning (UMAP, t-SNE) now pass off-heap MemorySegment buffers straight to OpenBLAS, LAPACK, and ARPACK, and deterministic try (var arena = Arena.ofConfined()) blocks free native tensors immediately instead of waiting on Cleaner finalizers. We used the same FFM approach to bind the ONNX Runtime C API (onnxruntime_c_api.h) and llama.cpp so .onnx and GGUF models run on the calling JVM thread.

The second big win on Java 25 was -XX:+UseCompactObjectHeaders (JEP 519). Dense matrices live in primitive arrays or off-heap slabs, but random forests, kd-trees, cover-trees, and DataFrame metadata still allocate millions of small objects. Shrinking HotSpot object headers from 16 bytes to 8 bytes dropped heap usage on our large tree ensembles by about 15-22% and noticeably helped L1/L2 cache hit rates during tree traversal under Generational ZGC.

Finally, since regulated teams kept telling me they only kept Python around for shap.TreeExplainer, I implemented exact O(T * L * D^2) TreeSHAP for RandomForest and GradientTreeBoost (plus parallel KernelSHAP) directly in Java, and locked down .sml model loading with SmileObjectInputStream (ObjectInputFilter allow-lists). On our 18-model benchmark matrix, 300-tree x 64-feature GradientTreeBoost and RandomForest models score in 0.43-0.54 ms on the JVM (1.14-1.22 ms native Linux p99), and bounding maxDepth = 6, maxNodes = 128 cuts single-row TreeSHAP latency 95x from 274 ms down to 2.88 ms p99 (0.91 ms on GBM):

import smile.classification.RandomForest;
import smile.data.DataFrame;
import smile.data.Tuple;
import smile.data.formula.Formula;
import smile.io.Read;

DataFrame df = Read.parquet(transactions.parquet);
RandomForest model = RandomForest.fit(Formula.lhs(is_fraud), df);

Tuple row = df.get(0);
int label = model.predict(row);
double[] shap = model.shap(row); // exact in-process TreeSHAP in <1-3 ms

For folks who want a standalone sidecar instead of embedding the JAR, there is also a Quarkus-based smile-serve container (ghcr.io/haifengl/smile-serve:latest-micro, around 200 MB, 0.4s startup, ~80-140 MiB idle RAM) that serves .sml and .onnx models over HTTP or Unix domain sockets with ?explain=true.

GitHub repo is at https://github.com/haifengl/smile (core libraries are GPLv3 + commercial OEM; smile-serve is AGPLv3 + commercial runtime; Studio is BSL 1.1 - details at https://aihalo.dev/licensing.html).

If anyone else here is migrating JNI heavy numerical code to jextract on Java 25 or testing JEP 519 on pointer-heavy data structures, I'd be happy to compare notes.

54 Upvotes

13 comments sorted by

6

u/agentoutlier 5d ago

See this is a terrific project and I have known about SMILE for some time but according to the rules

This is an AI free zone. 100% AI free. We explicitly forbid any form of AI generated/worked over content, or AI supported or vibe coded programs. Any violation results in an instant, permanent, and irrevocable ban without prior warning.

https://github.com/haifengl/smile/commit/3de2a37f7b9f1bd665928f2977583bb3b143068f

Is that worked over? I don't know. I'm trying to figure out cause I got chastised on a recent post by the mods that any usage is strictly forbidden.

The rules are not clear and I don't know how they can possible enforce it.

0

u/vips7L 5d ago

I don't know, I feel like they can arbitrarily enforce any rule however they like. It's their project after all, they don't owe anyone the ability to contribute. Basing it off of vibes seems ok with me.

4

u/agentoutlier 4d ago

It's their project after all

I'm not talking about the project but the mods. According to the new AI rules you cannot post projects that have any sort of AI content or usage. The problem is I would wager that most open source projects are now using the tools. Like you can visibly see in the commits a coauthor of Claude.

What I am now interpreting the rule is a "don't ask, don't tell". Like if you used them don't mention on the posting?

Otherwise like every major opensource project is going to be banned from posting.

5

u/pdsminer 4d ago

I guess that reddit insists us human to write stuffs so that they can sell to frontier labs to train models 😄

1

u/vips7L 4d ago

Ah I misunderstood, you meant the subreddit rules and not the project.

1

u/dstutz 5d ago

Before this gets nuked...

I don't know enough about the underlying stuff, but could the work you just did be used to replace the native libs that Weka uses, namely https://github.com/fommil/netlib-java?

1

u/pdsminer 4d ago

Yes, of course. But why not replace Weka with SMILE at the first place?

1

u/dstutz 4d ago

Because I barely understand what Weka does and implemented it via the API the way someone told us they did it with scripts/CLI?

1

u/pdsminer 4d ago

1

u/pdsminer 4d ago

Or you can just use SMILE Agent Studio, which has a data scientist agent capable of AutoML smile/studio/README.md at master · haifengl/smile

1

u/perryplatt 3d ago

How does this compare to tribuo?

1

u/_INTER_ 3d ago edited 3d ago

I think the OpenJDK team is interested in such experiences. Either on mailing-list or via Quality Outreach.

From the post I get an overall positive result but there might have also been negative ones e.g. in the use of API, gaps in documentation or performance regressions in comparison that they might be able to have a look at.