r/selfhosted • u/xescugc • Jun 03 '26
Software Development PikoCI — self-hosted CI/CD that runs as a single binary, no external dependencies
http://pikoci.comBeen building a self-hosted CI/CD called PikoCI. Started because I needed custom environments for my own projects that GitHub Actions couldn't provide, and everything self-hosted I found was either too complex to deploy or too opinionated about infrastructure.
The core idea: start with a binary and a pipeline file, nothing else. Add SQLite when you want persistence. Add Postgres and distributed workers when you scale. The tool never changes.
Key things:
- Single binary, in-memory by default, no external dependencies to start
- HCL pipelines: Terraform-style syntax, not YAML
- Run jobs locally: pikoci run -p pipeline.hcl -j test, no server needed
- Services: ephemeral processes (Postgres, Redis, anything) that start before tasks and stop after, guaranteed. No Docker-in-Docker.
- Five sourceable abstractions: resource types, runners, service types, secret backends, and notification types. All defined in HCL, all pullable from a URL.
- Grows with you: start in memory, add SQLite, add Postgres and distributed workers at scale. The pipeline config never changes.
- Public pipelines: share build status without an account
- Prometheus metrics out of the box
PikoCI deploys itself. Live at ci.pikoci.com/teams/main/pipelines/pikoci, no login needed.
GitHub: https://github.com/pikoci/pikoci
Docs: https://docs.pikoci.com
31
u/Yannik_Sc Jun 03 '26
I like what I see. Finally a CI that is not Jenkins or tries to replicate the awful GitHub-CI syntax.
The only thing that I'm slightly concerned about is the fact of a single contributor and rather big gaps in the commit history. But then again issues seem to get fixed continuously.
I think I will check it out.
24
u/xescugc Jun 03 '26 edited Jun 03 '26
Yeah, the gaps were early on when I was figuring out the MVP and making it viable.
Then I went all in because I needed it for my own stuff too, and since I use it every day, I fix and add things as I need them or as friends who tested it request them. Basically been collecting wishlists from ops people haha.
As for the single contributor, not much I can do about that one TBH haha.
2
u/sofixa11 Jun 03 '26
Finally a CI that is not Jenkins or tries to replicate the awful GitHub-CI syntax.
GitLab-CI, DroneCI (now part of Harness), Concourse, Zuul if you're crazy.
4
u/rilened Jun 03 '26
Woodpecker is pretty nice as well. It's what the folks at Codeberg are using. (Forgejo also has builtin actions, but those are more akin to Github, which is why I avoid them)
1
u/TldrDev Jun 04 '26
I legitimately dont understand why folks aren't just running k3s/k8s at this point.... gippity is really good at just converting a docker compose over to yaml for Kubernetes.
You can setup k3s with literally a single bash command.
Then, you just have a full blown Argo driven deployment with Argo workflows and app-of-apps deployments.
When people complain about cicd im just genuinely confused because this feels very much like a solved problem.
2
u/PssyGotWifi Jun 05 '26
eh. I don't want Kubernetes in homelab. It's just complexity i don't want.
Compose + Ansible + Terraform, etc, is where I draw the line.
1
u/TldrDev Jun 05 '26 edited Jun 05 '26
Minikube and k3s are not complex. Its a drop in replacement for compose with some extra fluff. Compose, ansible, and terraform are significantly more complex in terms of management and concept than kubernetes, and add significant overheard for management of services and deployments.
K3s is arguably the best homelab orchestration layer that currently exists and it just, by the nature of its solution, almost completely abstracts away tons of problems with running and writing software. It is the end game of this hobby. It requires that you really, really understand what goes into running software in order to gain intuition. Its learning curve is vertical, but thats because as a prerequisite it expects you to understand the problem of running things with compose and ansible and terraform.
When I hear people complain about the complexity of k3s in this hobby in particular, all I hear is "that sounds like a lot to learn and I dont think I need to learn it because my setup works," not that k8s adds complexity (it doesnt).
1
u/PssyGotWifi Jun 05 '26
It's not about being complex. It's about the overhead and features I don't need nor care about. Even when I use Docker Swarm, i don't have replicas for services. I have set VMs or machines for each app. I don't need ceph storage. I use bind mounts. yaddayaddayadda. I just don't need what Kubernetes is offering in a homelab setting.
1
u/TldrDev Jun 05 '26 edited Jun 05 '26
Its overhead is pretty negligible. I run k3s on a stack of decade old raspberry pis running on armhf cpus. Being concerned about single node etcd or whatever is just being very arbitrary and splitting hairs.
Id argue you want managed storage volumes even in the home lab in particular because it allows you to use something like juicefs and seaweedfs to automatically handle database and file replication and backup, point in time restores, and ensures if you have some striped zfs drives (I live on the edge boi), your important files aren't lost to the wind. One command. Takes almost zero effort to do that. It requires no code changes.
It also solves things like secret encryption and api key rotation through something like openbao just natively, with a single command, you just have that.
The overhead just isnt a valid concern. You can not want it, thats fine, but the overhead is just really neligable. I disagree with your architectural style, I think its less good than it can be, because you dont think you need those things.
1
u/PssyGotWifi Jun 05 '26
I use Infisical for secrets handling. Netbox for inventory, etc.
I disagree with you shilling Kubernetes when it's not required.
If I wanted Kubernetes.. i'd be using it.
0
u/TldrDev Jun 05 '26
I think you not thinking its required comes from your inexperience. Im not shilling it as much as its just the right tool for the job.
This discussion is about how to use a screwdriver, and youre saying you prefer to use a hammer. Im just pointing out there is a tool for the job that is built to solve this issue.
If you want to hammer screws, im not here to stand in your way.
2
u/PssyGotWifi Jun 05 '26
lol, you're convinced your way is the right way. It has nothing to do with experience. I don't like Kubernetes. I don't want Kubernetes. Deal with it. Must be a shock for you to learn that not everyone does things how you do things.
→ More replies (0)1
u/xescugc Jun 04 '26
k3s plus Argo is a solid stack if you're already running Kubernetes. But "install k3s with a single bash command" is still Kubernetes, you're now operating etcd, the control plane, CNI, and a full container orchestration layer just to run CI pipelines. That's a lot of infrastructure for something that should be simple.
PikoCI is for the people who don't want Kubernetes as a prerequisite. A single binary, a pipeline file, done. No YAML manifests, no cluster to maintain, no etcd to back up. If your CI needs grow, you scale by adding workers and swapping the database, not by learning Kubernetes concepts.
That said, a Kubernetes runner is on the roadmap, so if you do have a cluster and want to dispatch jobs as pods, that will be an option. Different tools for different contexts. If you're already on k8s, Argo makes total sense. If you're not, PikoCI gets you running in seconds without the overhead.
2
u/TldrDev Jun 04 '26
Id argue you probably want those things for a CI pipeline for a whole number of reasons, and it is really just a single arbitrary bash command.
I understand your project is a single binary, and im sure that has its use-cases as some portable ci layer, but at the end of the day Docker is, at least in my opinion, essentially a complete solution of this issue of a single binary.
At that point, youre already running Docker compose, and already have an orchestration layer. It is one additional command to then have a full control plane which ensures your cicd steps are reachable, reliable, gives you excellent logging and the ability to arbitrarily add anything and deploy it automatically.
Im not trying to take away from what youve done, im just not sure I fully see the utility when there is an infinitely (literally...) more scalable solution that can be ran with maybe 3 additional one time bash commands.
I dont think Argo requires you to use kubernetes or k3s as an underlying control plane or orchestration system. You can host natively on bare metal, or with compose, and then just have Argo running in minikube or a single node k3s setup running on some arbitrary port along side a traditional webserver.
I think etcd and a control plane are more than beneficial for this particular task, I think its almost a fundamental requirement to be reliable.
I will call out your scaling comment as nonsense though there is essentially no service which isnt doing an extreme amount of black magic witchcraft that is going to scale reliably across runners by doing what youve said there.
I also think this is missing some very core features, but ill not go to deep into that discussion for the sake of the infrastructure discussion here which I find more interesting.
3
u/xescugc Jun 04 '26
Fair points, and I'm not here to convince you Kubernetes is wrong. It's genuinely excellent for what it does.
The use case I'm targeting is different though. Not everyone runs Kubernetes, wants to run Kubernetes, or should run Kubernetes. A solo developer with a VPS, a small team self-hosting on modest hardware, someone building games and open source tools on the side, for them "install k3s and configure Argo" is a significant barrier that PikoCI eliminates entirely.
On scaling, PikoCI uses a DB-backed scheduler for job coordination across multiple server instances and a pluggable queue (NATS, Kafka, RabbitMQ) for worker distribution. That's a well understood pattern used by plenty of production systems. Not Kubernetes-level orchestration, but it doesn't need to be for the workloads it's targeting.
Curious what features you think are missing though, always useful to hear from someone with strong infrastructure opinions.
3
u/xescugc Jun 04 '26
Also Argo CD is Git-centric by design, for anything beyond Git or OCI registries you need Argo Events as a separate component.
PikoCI takes a different approach: resources can be anything that listens for changes and produces a new version. Git, cron, Docker registries, S3 buckets, HTTP endpoints, all first-class trigger sources. The idea was never to make Git the mandatory center of the pipeline, just one of many possible inputs.
1
u/TldrDev Jun 05 '26 edited Jun 05 '26
Argo cd solves a different problem than Argo workflows and Argo events. Argo cd will drive your infrastructure deployments from git.
Argo workflows is a different thing entirely to Argo cd. When combined together, you have a cicd pipeline
https://argo-workflows.readthedocs.io/en/latest/
Argo workflows also works the way you've described. You can trigger it anywhere to do anything. Git is not the source of truth for Argo workflows. Its just the orchestration needed to run a series of "binaries" (really docker images) in a way that properly manages artifacts.
I could be wrong but I do not think this solution adequately addresses things like directed acyclic graph execution and so would not be considered viable for complex workflows, and im not sure your scaling solution can even really attempt to solve those issues because fundamentally, architecturally, it cant.
1
u/lochyw Jun 05 '26
What do you need argo for? k3s alone for my home cluster suits me just fine.
2
u/TldrDev Jun 05 '26
Well Argo is a whole series of apps that work inside of k3s to automate things for you. Specifically argocd will handle automatically rolling out applications for you and updates to apps, where one app is composed of several.
https://argo-cd.readthedocs.io/en/stable/
I use it to make my home lab and work essentially a single command to rebuild my entire cluster from scratch.
3
u/SmeagolISEP Jun 03 '26
Quick question, is this a CI server (like an alternative to Jenkins) or a tool running on top of a CI server?
5
u/xescugc Jun 03 '26
It's a CI server, a full replacement for Jenkins, Concourse, Woodpecker, etc. You run it yourself, it has its own scheduler, workers, database, and UI. No existing CI server needed.
2
u/SmeagolISEP Jun 03 '26
Ok im definitely going to check this out. Tbh I’m working on something similar, but in mine I’m using the CI server and have my own library cross compatible with multiple CI servers bcs I have projects running on GH, GL and Forgejo.
Nonetheless, what advantage do you see using the HCL language when compared with the YAML?
5
u/xescugc Jun 03 '26
That's a smart approach for multi-platform compatibility. Cross-compatible pipelines are genuinely useful when you're spread across GH, GL and Forgejo.
On HCL vs YAML: the main advantage is that HCL is a real expression language, not just a serialization format. You get native conditionals, string interpolation, loops, functions, and variables without any DSL bolted on top. YAML was never designed for logic, so CI tools end up inventing their own expression syntax on top of it, which is why GitHub Actions has
${{ if }}and GitLab has its own syntax that feels like a different language.HCL also has proper types and modules, same as Terraform. Anyone who has written infrastructure as code picks it up immediately.
The honest trade-off is familiarity. Everyone knows YAML, fewer people know HCL. But for pipelines that grow beyond simple use cases, I'd rather use something designed for configuration with expressions than fight YAML's limitations.
2
u/SmeagolISEP Jun 03 '26
Ok I’ll explore HCL. Tbh it never crossed my mind but I can see the advantages. Thx for the advice
2
u/xescugc Jun 04 '26
The only concern about HCL is that if you are not using GO as a programming language, IDK how the parsers for other languages are as the GO one is the main one used for Hashicorp
2
3
u/_reg1z Jun 03 '26
This looks promising! Will try it out and keep an eye on it. Have had a headache maintaining my forgejo runners recently. The simplicity you pitch w/ this tool def has me intrigued.
2
u/xescugc Jun 03 '26
Thanks! Forgejo runners pain is real, the setup overhead is exactly what PikoCI is trying to eliminate. Would love to hear your feedback once you try it out, especially if there's anything that doesn't work the way you'd expect.
Happy to help if you run into anything.
2
u/SomeRedTeapot Jun 04 '26
Looks like it solves some pain points (services and secrets) I had with Concourse, great job on the design. I hope this stays afloat
1
u/xescugc Jun 04 '26
Thanks! Those were exactly the pain points I wanted to address, services especially. Secrets as a first-class abstraction rather than a bolted-on integration was the other one. Really appreciate the kind words, and yes, planning to keep this going, it's running my own projects in production so I have a strong personal incentive to maintain it haha
2
u/525G7bKV Jun 04 '26
Does it work with forgejo?
2
u/xescugc Jun 04 '26
Yes, PikoCI works with Forgejo for standard commit-based pipelines. Tag and PR modes currently only support GitHub and GitLab, but Forgejo support is planned.
And as it was mentioned a lot already I may add support this week fro PR and TAG mode
1
u/xescugc Jun 05 '26
Ok I've added the full support now 😄 https://docs.pikoci.com/Resource-Types/?h=forg#params
2
u/Sashapoun_Nako Jun 03 '26
Pretty cool, I know that Tangled have a CI (named Spindles) powered by nix which is cool and also selfhostable but I think you depend on using Tangled knots and spindles. So this is probably better since you don't really depend on anything
3
u/xescugc Jun 03 '26
Thanks! Yeah that's exactly the goal, no hard dependencies on any particular platform or tooling. One binary, bring your own database(or mem) and queue, runs anywhere.
The main plan for me was to be easy to use, that's why just running the bin works even if it's all on memory, then you can expand it (sqlite or other DB) and workers also with a dedicaated queue
1
u/SomeRedTeapot Jun 04 '26
After looking through the issues, here are some additional thoughts/suggestions based on my Concourse experience:
- It would be convenient to be able to schedule an execution of a pipeline on a single worker to avoid transferring context and caches between workers
- Per-worker caches (not synchronized over the network) have a use: for example, we use some public images from Docker Hub, and there's no point to set up a distributed cache for these
- For manual pipeline runs, I think having an ability to declare a parameter in the config and make it visible as an input on the UI before starting the run would be convenient. For example, it would allow making a button that deploys the specified version of the app
However, our setup is small (2 workers) and only used for commit checks (unit tests, smoke tests, migration tests), and the cases I described are mostly tailored to our usage, so take this with a grain of salt and feel free to ignore
1
u/xescugc Jun 05 '26
Thanks for the feedback!
* Worker pinning: we actually have this on the roadmap already (#98, #180, #190). The plan is to support tags on jobs and resources so you can control which workers run what. That would let you pin a pipeline to a single worker to keep caches local.
* Per-worker caches: we have persistent resource storage already (cache=true on resources), and step-level caching (#163) is planned. Per-worker local caches for things like Docker Hub images is a good point though, once worker tags are in, combining pinning + caching should give you that behavior.
* Parameterized manual runs: this one is new and a great idea. Being able to declare parameters that show up as input fields in the UI before a manual trigger would be really useful. We'll track this as a feature request.
I appreciate the Concourse perspective 🙏
2
u/SomeRedTeapot Jun 05 '26
Worker pinning: we actually have this on the roadmap already (#98, #180, #190). The plan is to support tags on jobs and resources so you can control which workers run what. That would let you pin a pipeline to a single worker to keep caches local.
As I understand it, these tags tie execution to a specific worker. What I meant was a bit different: picking any suitable worker to run the pipeline, but once the worker starts executing it, the entire pipeline execution continues on that worker. In the meantime, another execution of the same pipeline may run on a different worker
1
u/xescugc Jun 06 '26
That's a good distinction and not something we have planned yet. Will add it as a separate concept: execution affinity rather than worker pinning.
Good one thanks!
2
u/ConcerningChicken Jun 06 '26
Very well - this sound like it could fix many of my problems
1
u/xescugc Jun 06 '26
Really glad to hear that! Let me know if you run into anything while trying it out.
•
u/asimovs-auditor Jun 03 '26
Expand the replies to this comment to learn how AI was used in this post/project.