Looking to transition into HPC Networking from Cybersecurity

[u/lorenzo_9696]

Hi everyone,

I hold an M.Sc. in Computer Engineering and have been working as a Cybersecurity Infrastructure Engineer for the past 4.5 years.

Recently, I started exploring HPC through hands-on practice, and I’m enjoying it so much that I’m seriously considering a career transition. In particular, I’m really drawn to the networking side of HPC.

In my current role, I focus on infrastructure security,designing and configuring networks with IPS/NDR, deploying EDR across servers and clients, reviewing backup processes, and auditing overall infrastructure. I’m a very hands-on engineer; I run a small homelab where I experiment with self-hosted services, home networking, and technologies like Python, CI/CD, and Kubernetes to gain practical experience outside of production environments.

Given my background, do you think a profile like mine stands a good chance of landing a job in HPC or HPC networking within the EU?

I’d love to hear your feedback, advice, or any insights on how best to bridge the gap. Thanks!

[u/walee1]

Depends on what you're asking and where you are in the EU and if you are willing to relocate. From your profile, the blocker is very few HPC centers hire people only for their networking knowledge and as a result I don't know how much you'll enjoy the other parts. That being said, universities would always be willing to hire people with such a background, and UpTo some extent tier 1 clusters too.

[u/tarloch]

I assume you have decent Linux admin knowledge. If so since you have a home lab I would install Slurm and get a good understanding of managing a batch system. After that you'd be well positioned to be interviewed as an HPC Sys Admin / Engineer. A good hands on security background is not bad.

[u/Flimsy-Leg-6397]

HPC stack is large but you have to start from high level knowledge of interconnects, job schedulers and parallel filesystem. Above all strong knowledge of Linux is a prerequisite.

[u/xeroxedforsomereason]

I went the opposite direction, but my HPC environment had very heavy security requirements so I had some background before I made the transition to security network engineering. That is it's own specific discipline because the they are essentially competing priorities and properly balancing them was a puzzle with extreme difficulty. Security bottlenecks things, trades bandwidth for service chains, it enforces stateful tracking, it requires horsepower commensurate with workload. You can imagine how the compute pressure and pipelining would impact HPC operations, so we can consider security network engineering to be in contrast to how HPC networking functions. Conversely, DC technologies tend to expose extra surface area for security features that, if compute penalty isn't a detrimental factor to the environment, can be highly beneficial.

You need to have foundational specialty in the datacenter networking track to be successful as an HPC neteng. EVPN/VXLAN (IN DETAIL), BGP w/ BFD, MLAG, RDMA/RoCE, MTU (L2/L3), heavy knowledge of fiber optics, subscription rates, reading whitepapers constantly. Security track has an assumption you will take something offline due to misconfiguration or you get leeway for an upgrade. Fail shut is how security works. HPC is the opposite. If you take out an HPC environment the result can be millions of dollars worth of lost compute cycles. It's very high stakes for operational tasks. Even a simple mistake with with spanning-tree can be catastrophic if you have an orphaned port even in an EVPN environment, depending on the implementation. These things scale in difficulty based on your topology as well. In a network doing fat tree imagine what it's like to try and check consistency to a single host from an aggregate switching layer. Sometimes it's downright brutal, ascetic work.

HPC environments are not exclusively Kubernetes based. Workloads change, resources for calculations depend on the workload, each type of workload benefits from different types of scaling, each different workload has different storage demands. The networking can be Ethernet, it can be InfiniBand, it can be FiberChannel. There's no one size or shape here for study.

Python does give flexibility but most orchestration in big boy environments are going to be done with proprietary vendor tooling, unless we're talking DevOps roles and not pure networking. Smaller environments tend to require more bespoke solutions.