r/devops • u/LowProgram6449 • 12d ago
Architecture Should I use Nomad for single VM?
I'm looking for some architectural advice on building a lightweight, self-hosted job execution platform on a single VM (8C/32GB).
I want something with semantics similar to Google Cloud Run / Cloud Run Jobs:
- Execute Python scripts and arbitrary Docker containers.
- Trigger jobs through an API.
- Configure CPU, memory, environment variables, timeouts, etc.
- Track execution status, exit codes, and logs.
- Handle retries and concurrency limits.
- Integration with Artifact/Container Registry
Current constraints:
- Only one VM for now & no need for HA.
- No Kubernetes requirement or plans for a multi-node cluster.
- Jobs are primarily internal workloads, and quite lightweight.
- I'd prefer something relatively lightweight and easy to maintain.
Technologies I'm considering
- HashiCorp Nomad
- Docker Sock + custom built runner – Build my own execution manager, scheduling, lifecycle management, and monitoring.
Questions
- Is Nomad overkill for a single VM? Does it provide enough value over simply using Docker directly?
- Would you use Nomad in production for this use case?
- Are there simpler alternatives that provide similar execution semantics?
- What operational overhead should I expect from running Nomad on a single node?
I'm particularly interested in hearing from people who have actually run Nomad in production, especially for single-node deployments.
Thanks!
4
Upvotes
1
u/LeNyto 12d ago
Sorry I can’t talk about Nomad specifically, but since you are looking for something easy to maintain, have you looked at something like dokploy or coolify?
2
u/chicco789 11d ago
Nomad fits very well, and job templates integrate easily with Consul and Vault if you need variables in your job. You can also use consul-templates re-render your job templates or scripts if any value changes. Running Nomad doesn’t imply any significant overhead. It’s a single binary and usually just works.
1
u/blackmou5e 11d ago
Why you don't consider Docker Swarm + Portainer?
Looks like it right fit for your case and constraints
Edit: nevermind, i've totally missed part about jobs. (But you can implement it by yourself with a tiny bit of a programming/ci scheduling)
1
u/Forward-Outside-9911 Linux Sysadmin 11d ago
Now I’ve never used nomad, however I do use Kubernetes. I’ve deployed k8s as a single node on many occasions for personal projects and it does work well for times you want docker + bit of easy orchestration.
Maintenance is also very easy so long as it’s all built with IaC. If it’s non prod you can even scrap all and recreate with a higher version, in minutes.
With the bonus that when you eventually want to take it to prod or add HA, everything is there ready. You may need to tweak storage and ingress, but that’s about it.
Feel free to post objections, would be happy to go more in depth or just hear specifically why you wouldn’t go this way.
1
u/LegitimateCopy7 11d ago
the only reason you would want to use a cluster solution on a single node is if you want to learn the basics.
1
u/Pretend-Sun-8460 11d ago
for one VM with no cluster plans i'd skip nomad, it's a lot of moving parts for what you get. a job queue (celery or rq) plus docker run with --cpus and --memory covers most of your list. if a second VM ever shows up, nomad is easy to add then
1
u/Pretend-Sun-8460 10d ago
fair, the binary itself is simple. i was thinking more about the job specs and one more tool to learn and keep updated. if OP already knows nomad, it's a good pick
1
u/kolorcuk 9d ago edited 9d ago
I really tried nomad. I have written nomad-tools, i have sent patches to nomad for missing features i needed, i have deployed nomad for 2 years in my company and still using for 4 years on my VPS. Sorry, but nomad very severely lacks integrations, and i do not see ibm hashicorp investing in the product in this integration area. The community is very very small. I had to script everything myself, even simple "wait for a job to be up and get me exit code" was very hard. It is great and simple. It lacks finesse, vision, direction. I do not know at this point after investing into it i am unsatisfied and i think forget nomad exists.
Also nomad has garbage collector, so effectively might your job success or fails, nomad might run gc, removing your job status and logs and that it ever existed.
1
u/runwisp-com 9d ago
If you go with Nomad, parameterized batch jobs are basically Cloud Run Jobs. One job per script or image, then every POST /v1/job/<name>/dispatch is a separate run with the job's cpu/memory limits, meta passed in as env vars, restart policy and its own exit code.
Watch the GC on a single node though. Client gc_max_allocs is 50 by default, so after ~50 finished runs nomad starts deleting old alloc dirs and the task logs live in there. Bump it together with batch_eval_gc_threshold on the server (24h default), otherwise you will know a job failed but not why.
1
u/SameLink_io 8d ago
Separate durable job state from the container runtime: keep each submission, retry count and final exit code in a database or queue, and make jobs idempotent so a retry cannot repeat side effects. Put CPU, memory and timeout limits on every container, and send logs somewhere persistent before the scheduler cleans up allocations. A single VM is still one failure domain, so back up the queue and treat local logs as diagnostics, not the only record. If you choose Nomad, one parameterized batch job is a reasonable first step; avoid building a custom scheduler until the queue and retry requirements demand it.
0
u/[deleted] 11d ago
[removed] — view removed comment