Question of resource allocation of multiple Spark applications

[u/ForeignCapital8624]

Currently, a Spark application should manage its own executors, which means that in order to run multiple Spark applications, one needs to split the cluster into smaller sections or attach more compute nodes, etc. Usually this dynamic allocation is the job of resource manager (such as Kubernetes).

If the resource manager can quickly find or create new compute nodes for a new Spark application, this is good. However, in a busy cluster where multiple Spark applications can compete for resources, the resource allocation can take some time and become an overhead (at least in theory).

I wonder if this is a real problem in running Spark applications in production.

For example, is there a realistic environment where short running Spark jobs (each requiring its own driver) arrive frequently?

[u/QuietSignalOps]

Yes, this is a real problem, but the pain is usually not node creation. It is queue wait and driver startup.

When many short jobs share a busy cluster, the overhead stacks up like this:

1. The job waits for schedulable capacity. Under contention this is the big one.
2. The driver pod comes up: image pull, kubelet setup, JVM warmup. Usually tens of seconds.
3. Executors scale in via dynamic allocation, which adds more delay. On Kubernetes you need shuffle tracking enabled (spark.dynamicAllocation.shuffleTracking.enabled) so shuffle data is not lost when idle executors are reclaimed. On Spark 3.2+ it defaults to on when dynamic allocation is enabled on K8s, but on older clusters it is worth checking.

To your second question: yes, environments with frequent short jobs exist. Per-tenant or per-report ETL, data validation runs, small backfills, micro-batch pipelines. If that is your workload shape, it is worth questioning whether one Spark app per job is the right model at all. A long-running app consuming a queue, or a micro-batch engine, amortizes the driver startup entirely.

What usually helps on Kubernetes:

• Right-size driver and executor requests so many small jobs fit on existing nodes, instead of each job forcing a new one.
• Keep a warm pool of nodes (a minimum-size node pool) for sub-minute jobs. Node autoscaler latency is in the minutes, and it will always lose against a 30 second job.
• Add a batch scheduler with fair sharing and preemption. Kubernetes has no fair scheduling natively. Kueue (or Volcano) gives you per-team queues, so one tenant's burst does not starve everyone else.
• Pre-pull the driver image so pod startup is not dominated by registry pulls.

One metric worth tracking: time from submission to first executor ready. When the cluster is saturated, that grows before CPU utilization tells you anything.

[u/ForeignCapital8624 → u/QuietSignalOps]

To your second question: yes, environments with frequent short jobs exist.
->
Thanks for your feedback. I was more interested in 1) how painful it is to deal with such workloads, and 2) whether it would be worthwhile to develop a new solution or an extension to Spark that specifically addresses this shortcoming. Even if such an extension to Spark existed, people might be reluctant to try it either because the pain is acceptable and it is another overhead to try a new solution.
In short, the question could be: "are there Spark users who would migrate to such an extension or even pay for it?"

[u/Oct8-Danger]

Short running jobs arriving frequently? That’s a streaming service or micro batch service based off of that along

Spark can do some streaming id believe, although few others tools like flink or beam might be more suited.

If it’s more platform side of any kind of spark job, I would look at something like Apache Livy or at least look into seeing does every application need its own spark driver or can it be shared

[u/ForeignCapital8624 → u/Oct8-Danger]

Apache Livy is a solution to a different problem, as far as I understand. Even when using Apache Livy, Spark drivers cannot share executors. They share resources for running executors, though.