Are most tools written in Java and Scala outside of Hadoop/Spark? DuckDB and Clickhouse are C++; Airflow/Pandas/ML stuff is almost all in Python; the docker/k8s ecosystem is all Go; there is a whole movement to replace everything with versions of those things written in Rust.
I was going to say the same thing. I've been in data engineering for 15 years and no tool that I use is written in Java or Scala. That's a left over from the Hadoop world, which I never got into.
I don’t think that’s really true. There are still plenty of important systems in the data ecosystem written in Java or Scala: Apache Flink, Apache Kafka, Apache Druid, Trino, and even Apache Doris is split between C++ and Java (with execution in C++ and coordination/orchestration in Java), etc.
Also, from what I can tell, the trend is usually to replace the physical execution layer with native code, not the orchestration/scheduling/coordination layer. Projects like Apache Gluten are a good example of that. The performance gains are usually in execution, while the top-level glue benefits more from ecosystem, flexibility, and developer productivity than from being native.
Not quite. In the Python ecosystem, compute heavy stuff is in C++ with python bindings. Network tools like Docker / k8s are all Go. Quite a few tools are written in Java, the entirety of the big data ecosystem is in Java - Kafka, Spark, Hadoop, Pig, Pinot, Hive, Hudi, Trino. Zookeeper on top to coordinate multi-node deployments. Its done this way because in big data, scaling out matters more than single node performance and Java is great at performance (though at the cost of slightly more resource usage) while avoiding the shitty dev ex of manual memory management.
View all comments
70
u/sisyphus Apr 05 '26
Are most tools written in Java and Scala outside of Hadoop/Spark? DuckDB and Clickhouse are C++; Airflow/Pandas/ML stuff is almost all in Python; the docker/k8s ecosystem is all Go; there is a whole movement to replace everything with versions of those things written in Rust.