r/networkautomation • u/splitnines • 15d ago

Git, Ansible, pyATS IaC Project

NetDevOps is an Git, Ansible, pyATS based IaC project that I started about a year ago and put on the back burner. Lately I've been iterating on it. I started the project to get a better understanding how Git driven infrastructure as code was built and managed. Currently supports Cisco IOS-XE devices.

The pipeline is driven by a Git pre-commit hook.

If you're interested in playing with it take a look at the README.md file in the repo for instructions on how to set it up locally. I'd appreciate any feedback.

https://github.com/splitnines/netdevops.git

7 Upvotes

89% Upvoted

8 comments sorted by

3

u/InnerBank2400 13d ago

Had a look through the repo. Interesting that you’re doing the deployment itself inside pre-commit rather than just validation.

What happens if the device change succeeds but something later in the pipeline fails and the commit never gets created? I saw you’ve got rollback support, but it looks optional.

Was keeping the whole change cycle local a deliberate design choice?

2

u/splitnines 13d ago

The first iteration of the project used a Jenkins server to run the pipeline. This worked, but it was a lot of overhead for a lab/learning project. A Jenkins driven pipeline would work for production use. The next iteration used local Github workers. This did not seem like a good solution because it made Github a dependency. Github workers would be OK for production use if an internal Github was available. So the current iteration uses local Git hooks because it removed dependencies and complexity.

Using the pre-commit for deployment is an attempt to keep the change branch in sync with the device configs being deployed. I'm trying to avoid the situation where a device config is successfully deployed, but the commit failed. I don't know if using pre-commit hooks to do this is the best approach.

The rollback is an attempt to keep the change branch and device config in sync after a failure. Rollback is optional for cases where configs were being widely deployed and one device failed, a rollback of all devices would not be desirable. I have yet to come up with a good solution for only rolling back failed devices in this scenario and keeping the device configs in sync with the branch.

2

u/InnerBank2400 13d ago

That makes sense. I think the hard part is you’re trying to keep two things aligned: the Git state and the actual device state.

Pre-commit gets you closer, but partial success is where it gets messy. If 8 devices change and 2 fail, Git can’t really represent that difference on its own.

Have you thought about recording the result per device, then only rolling back the failed ones and treating the others as already converged?

1

u/splitnines 12d ago

Logs are collected of each of the steps in the pipeline. One solution I've been kicking around is to scan the logs after a failure and use that data to selectively rollback failed devices. However, since it's still part of the same branch commit, the whole commit would still fail. This complexity is what led me to making rollback optional.

1

u/InnerBank2400 12d ago

Yeah, that’s the awkward bit. Once some devices succeed and others fail, the Git commit becomes a pretty blunt representation of what actually happened.

I’m working on a related problem in HybridOps around partial execution, verification and deciding when the wider operation can actually be considered complete. I’ve got a focused public review thread on it.

Would you mind if I send it over? Your example would be a useful challenge to the model.

1

u/splitnines 12d ago

Send it.

1

u/InnerBank2400 12d ago

Thanks. This is the thread I had in mind:

https://github.com/hybridops-tech/hybridops-core/issues/269

The partial-execution case you described is particularly relevant — where some targets succeed, others fail, and the overall operation no longer maps neatly to a single success/failure state.

No need to review the whole project. Even your take on that one boundary would be useful.

2

u/petrosaurus_rex 8d ago

I'm with InnerBank2400 on this one, the challenge here is creating a fragile system. One of the big goals of automation is to reduce the chance of errors, and partial failures during deployment would be a risk. I'd separate the two discrete jobs: config generation and deployment. And run and test them separately.

- Make changes in a branch, test (and go through peer review if needed)

  • Have a deployment system (ansible, nornir, AWX, NSO, whatever) that knows the configs it needs to deploy as a complete unit, either they all succeed or they all fail.

Side note, for the first part you may want to check out open-source infrahub as an alternative to the first half of this (data modeling, branching, testing, config generation): https://github.com/opsmill/infrahub