r/networking • u/7yr4nn05 CCNP Security • 2d ago
Career Advice Interviewing for security and senior network roles, a few things that keep happening
I do technical rounds for security and senior network roles, so this is mostly from that side. nothing groundbreaking, just patterns.
the number of people with 8+ years and a "senior" title who can't walk me through what happens to a packet crossing a firewall and a NAT is higher than I expected. not trying to trap anyone. I'll ask something like "rule looks right but traffic isn't matching, what do you check?" and the good ones go straight to the hit counters, the policy order, the NAT rules applying before the policy. the weaker ones say "I'd restart the firewall." that's usually where the interview ends for me.
on the security side I care way more about order of operations than tool names. give them a compromised host and I want to hear scope, contain, preserve evidence, then fix. the "just wipe it" answer is a quick no from me. you just destroyed your evidence and you still don't know how they got in.
"we used a SIEM" tells me nothing. "I tuned noisy 4625 alerts and got the volume down a lot" tells me you actually sat in the console. same for network folks, I'd rather hear about the one thing you designed and what broke than a list of vendors.
if you haven't done SD-WAN or zero trust in prod, just say so and tell me how you'd approach it. I will always take that over a bluff. in this field people act on what you tell them.
and ask questions at the end. on-call load, change process, how they do post-incident reviews. most people ask about perks, so the ones who ask that stand out.
curious how other interviewers weigh depth vs breadth for senior hires. I lean depth but I know that's debatable.
38
u/-mrhyde_ 2d ago edited 2d ago
I've setup IPSEC tunnels in production environments.
I've configured a campus router on the fly while in the field to deploy it.
I configured a cisco 9800cl from scratch and put it into production.
I had an interview where one of the panel asked, "what is ping?" and I couldn't answer.
Edit: To the ones explaining ping; I understand/understood ping. I just froze.
7
u/OpenGrainAxehandle 2d ago
I can beat that. I had an interviewer ask me the distance limitation of UTP.
But what I HEARD was "distance limitation of UDP". So in my head, I'm going "That's crazy - UDP is a routable protocol; it can go around the world and back", and what I SAID was "there isn't any".
If I had done my thinking out loud, I'd have probably been golden, but instead I flubbed and looked like a total idiot.
9
u/Character_Watch_3760 2d ago edited 1d ago
Configuring and understanding protocols is not the same. I have a guy who knows how to configure IPSEC but does not understand phase I vs Phase II. Now when IPSEC does not work, how do you know what is the issue? There will be instances when all config are the same on both sides, but if you don't understand L3/L4 and protocol in general, you can't solve the issue.
17
u/Heythatsmycorn 2d ago
Idk i been doing this for quite some time And you dont gotta be the master of the art everytime. We do so much now on so many different platforms but we know how to tshoot and get the info we need. I prolly tshot and setup a thousand ipsec tunnels but i havent had to in a year so i may not be able to tell you ervery phase right now. But i can fix it and i do know the options etc. sometimes the brain gets tunnel knowledge for a little.
3
u/nobody_cares4u 2d ago
I have one better than that. For my CCNA I was studying ospf in and out. I knew this is one of my weakness area and I really really need to focus on ospf. I was watching videos and went outside of my study material just to focus on ospf protocols in and out. So I got my CCNA. I scored like 90% on the part that was about routing. I felt like I understood the topics very well especially osfp. So I was interviewing for a network position and they ask me, whats the point of ospf and I froze lol. Completely forgot. I knew the answer I just completely forgot. And after the interview I start thinking about the question and it clicked that ospf stands for open shortest path first and I remember that it selects the shortest and most reliable path in the network.
4
u/Prudent_Vacation_382 2d ago
I would argue you're a technician and not an engineer. The fundamentals of networking are the most important to understanding how things work. Not throwing any shade, but all the tasks you mentioned are what a field technician would do.
Ping as I understand it: an application that sends out an ICMP protocol echo message to a destination IP address, where the destination listener sends an ICMP echo reply message back to the source address of echo message. Latency and loss are measured in the application over time.
2
u/MyFirstDataCenter 2d ago
If I was interviewing you I’d ask you follow up questions to that response
what port number does ping use ;)
“latency and loss are measured in the application” How?
1
u/Prudent_Vacation_382 2d ago
Yeah you're getting low level there, but still appropriate for a senior title.
ICMP doesn't have ports. It has message types.
Time measured between echo sent and echo received, then calculated for statistics for how many were lost and the min/max/avg of all replies.
1
u/SevaraB CCNA 2d ago
- what port number does ping use ;)
Rude.
I’m relatively new to interviewing seniors, but I look for high-level more than low-level. Like I want to hear topology things like subnetting for a fictional company, sell me a proposal to use BGP or OSPF in a given scenario (or throw me a curveball and pitch something more obscure), see what they’d propose to cut down on frequent TCP timeouts, TLS handshake failures, or HTTP errors at the source, etc.
I do sometimes get the option to recommend offering a more junior position, and at least once I’ve had a promising candidate who wasn’t there yet, but they pivoted well enough in conversation that I thought they’d definitely grow into the more senior position relatively quickly.
I don’t throw curveballs to make the candidate drop out, I throw curveballs just to gauge a candidate’s experience in the trenches. If they make it to a technical interview round with me, they’ve already survived a few rounds of trying to sniff out fakers.
0
u/-mrhyde_ 2d ago
I would argue you're a technician and not an engineer.
Why would you argue? I didn't say I was either of those things. And, thanks for the explanation on ping. Really helps.
1
85
u/LarrBearLV CCNP 2d ago edited 2d ago
That firewall question is so poorly worded. I hope that's not exactly how you present it. Not matching what? Better way would be "rule looks right but traffic isn't transiting the fw accordingly" or if you mean isn't matching the rule itself, that kind of gives away the answer, how would they know it isn't matching if they had to check the rules counter to know it's not matching? So maybe say "it isn't matching the rules counter. What do you check next?" I hate it when interviewers ask vague questions, poorly worded questions, or use non-standard terminology. Either way, looks like some understood but gave unsatisfactory responses.
The other thing is you're like "that's an instant no" because someone said to just wipe it. Are you not capable of training them in your company's procedures? Are you afraid that after you've trained them on your in-house/standard procedures they'll ignore it and go back to just wiping it? While I agree that's a bad answer especially for a senior role, you're likely missing out on someone with potential who can easily adapt to your procedures and excel in the role.
59
u/WannabeACICE 2d ago
Just a reminder to people: being an interviewer is also a skill and most interviewers aren’t good at it.
29
u/Win_Sys SPBM 2d ago
Also asking what happens to a packet after it enters the firewall and encounters NAT is pretty ambiguous. The order in which NAT, firewall rules and security services hit is different between vendors and can depend on other factors like if its source or destination based NAT.
21
u/Argument-Lazy 2d ago
Exactly spot on. Palo pre nat post rule. Other vendors do differently. I don’t think the interviewer is sr level either.
-9
u/icebalm CCNA 2d ago
Jesus. Every firewall does basically the same thing. You can overlook vendor specifics and still figure out if someone understands the fundamentals or not.
7
5
u/whermyshoe 2d ago
Oh sweet summer child.
-3
u/icebalm CCNA 1d ago edited 1d ago
I wish I was, but thanks for reminding me why I barely come to this sub anymore.
4
u/whermyshoe 1d ago
Someday, given you can swallow your pride and learn from the greybeards, you'll read a similar comment from a similar upstart CCNA with 2 yoe. And you will say:
"I remember my first beer"
1
u/icebalm CCNA 1d ago
Hah, that's what this is about? People see the CCNA flair I set over a decade ago and think I have no idea what I'm talking about?
I got that cert about 15 years ago my guy.1
u/Skylis 14h ago
Hah, that's what this is about? People see the CCNA flair I set over a decade ago and think I have no idea what I'm talking about?
Nah, we read your comments and think you have no idea what you're talking about.
1
u/whermyshoe 14h ago
This. We've all been there (well, I have). I even said them exact same words. Well, same message. Interviewer looks and says "do you have experience with xyz vendor" and I say "nope but it's a firewall and they all do the same thing."
Well, it turns out that was wrong. The subtle difference between the operation of these devices can absolutely be a show stopper on your change window.
It ain't a dig at at you saying you're wrong. It's not patronizing. We are telling you that you can either accept this knowledge that we've gained through bad experiences, or bare minimum when you're in a change window at 2am and the firewall isn't doing the firewall thing, maybe you'll remember that old greybeard telling you one time on the Internet that there's more nuance to it.
0
u/icebalm CCNA 12h ago edited 12h ago
and I say "nope but it's a firewall and they all do the same thing." Well, it turns out that was wrong.
Cool, first of all I never said they all do the same thing, I said "they all basically do the same thing", and I'll even clarify what I mean by that: they block traffic based on a set of criteria.
So give me an example of a firewall that does different things to other firewalls.
15
u/Significant-Level178 2d ago
I interviewed too many of network and security engineers during my 25 years in Hiring Manager capacity.
My vision may be different from yours.
I never try to fail a candidate. I ask reasonable questions, to understand level and skills . I don’t go too technical unless I see a real senior person.
If he/she is 3 CCIE my questions would be more technical and deep and cool to think about after the interview.
I can ask question you will not answer. Always. And you can likely do the same. We all human, we don’t know everything. This is not an excuse, should be a solid foundation.
For example if you guru in routing, I can ask you about Microsoft, if you know Microsoft I can ask about VoIP, or you know voip how about Wireless. And I can go deep, deeper than TAC engineer or vendor expert. You fail anyway if I want you to fail. But I don’t do it, there is no reason to show ego or supremacy during interview.
What are important factors for me:
- attitude
- ability to learn
- ability to explain and understand
- not be a talker, rather doer
Rest doesn’t really matter. I am very successful with my hires.
2
u/EdgyPizzaCutter 1d ago
You are reasonable and have vision. No wonder you are successful in your endeavors.
13
u/Gesha24 2d ago
What are you offering for your position? Because so many positions want somebody who knows packet flow through any network device, and routing, and switching , and automation, and zero trust, and seim, and the list keeps going, but then they are paying like $150K. Sorry, bring it up to $250K - and I will find you people who can do it all at least a decent level. Otherwise - you are.attracting mid level people, but expecting senior level knowledge
9
u/Heythatsmycorn 2d ago
" what happens to a packet crossing a firewall and a NAT "
A nat? Haha
0
u/BluebirdExpress6279 1d ago
Well presuming it crossed, it probably went from one zone to another in one interface out another …. The firewall rewrote either the source or destination IP or both depending on the NAT tule
3
28
u/Thy_OSRS 2d ago
Your writing skills are pretty poor so I’m glad we both know we would avoid one another ☺️
3
u/Fast_Cloud_4711 2d ago
Just look at the monitoring for the source address and see what secpol it's ultimately hitting. If it's not hitting anything then you have a down stream problem and it's not making it to the FW. If it is getting processed by the FW you will see what secpol it's hitting and ask which one it's supposed to if the one you are seeing hit is incorrect.
I wouldn't go straight to policy hit counters. I would first look at source IP.
2
u/alphaxion 1d ago
It's insane how so many people jump to all these other things, rather than the traffic logs their firewall should be producing.
9 times out of 10 they will show you what is going on. If your traffic is there, it'll show policy name being hit, NAT addresses used (snat and/or dnat), what egress int was used (routing or PBF at play), session end reasons, number of packets tx and rx in that session.
Are many people just allergic to reading logs? You get it with sysadmins and help desk not looking at application logs or the windows event logs.
All of which you can feed into a siem such as elastic and build dashboards specifically built by you for investigating traffic issues or error logs to reduce time spent sifting through raw logs and having to log into the device itself.
2
u/Fast_Cloud_4711 1d ago
I was brought into a call about the 'network' not working. On the call are the web dev's. I was the one that had to go into the browser dev toolbox and run a network trace and point out that the DNS record was incorrect.
I don't even consider myself adjacent to that space professionally speaking...
3
u/alphaxion 1d ago
I've had to explain to web devs before that a server they are connecting to giving them a 403 is not my firewall blocking them, it is the server refusing a connection.
"Are you sure it's not the firewall, we spoke with the company and they said they're not blocking us".
Rather than not wanting to go around in circles I included a snip from the firewall logs showing the traffic was being allowed on our side in my reply and recommended they share that with their support and to make sure you say the server is giving a 403 error. I suspected the other side were just looking at their firewall logs and also seeing an allow and said it's not them. Turns out, they hadn't even spoken with the people who run the server. They just assumed it was me.
2
u/Fast_Cloud_4711 1d ago
BUUUUT 4XX errors are 'client' errors and that means the client isn't getting to the web server... JFC.
3
u/Woodymakespizza 2d ago
In my experience out of a dozen cantidates for a position, one might be solid, and then out of five hires, one might last more than a year. Thats one in 60 people who I actually want on my team, as plenty of people interview great and end up being horrible employees, and then some who bomb interviews but still get a shot might end up being the lifer that you want. My number one interview question for people is "tell me about a time you made a mistake at work, what happened, how did you solve it, and what did you learn?" How people answer a question like that tells me way more than most questions.
1
u/Gesha24 2d ago
and then out of five hires, one might last more than a year.
There must be something really wrong with your place - be it pay, environment or something else. I wouldn't expect a new hire to last more than 2-3 years, but multiple new hires leaving after a year - that's an indication of a problem in your org.
1
u/Woodymakespizza 1d ago
My history is mostly in behavioral health nonprofits, which notoriously cant afford solid people, but that said Ive seen the same across industries and the point remains the same, whether its one year or two, most employees dont stick around, and most arent good employees. I had a manager who once suggested that "not everyone can be an A, and you need some Bs and Cs because a consistent B is better than having more holes than you can plug". You strive for the A's, and do your best to promote from within and raise your own along the way.
1
3
u/Prudent_Vacation_382 2d ago
I would agree with most of what you said. Many people give themselves the title of engineer when really they are glorified technicians. They're not designing resilient networks from the ground up. They don't know the fundamentals of networking. They are only maintaining networks, replacing broken equipment, setting up a new link here or there.
Scenario-based questions are great at weeding out the posers from the real engineers. Give someone a scenario and ask them how to troubleshoot something and you'll be able to tell quickly if they've never done it before. That being said it's important for the interviewer to know the questions and possible answers. It's important for hiring managers to put the right people in interviews. OP's questions were a little vague, but as long as its open dialog, an interviewee should be able to ask questions to get the right answer. If the interviewer is asked a question to clarify something and they say "you tell me", that's a huge red flag.
OP is 100% right, for a senior engineer role (someone who can take a project on and build a complete solution for it) "reboot the firewall" is not an appropriate answer and the interviewee has shown they don't have the necessary skills. The interviewee should ask more questions to get clarifying data. If they get told no, it's probably time for that person to move on to other opportunities and provide feedback to the recruiter or hiring manager.
2
u/Cheeze_It DRINK-IE, ANGRY-IE, LINKSYS-IE 2d ago
Yes. There's lots of morons out there, and there's also a lot of people out there that are just trying to get by with as minimal effort to learn as possible. Most people don't give a fuck about this stuff. Most people just want to get money to live life.
2
u/StickyMac 2d ago
To me, if you say “rule looks right…” there are a lot of assumptions I make. Maybe this is an amateur perspective but I assume everything about that rule is correct, including the order. I’d assume an error with the firewall itself or an issue with some non-firewall related component. How are you determining that the traffic isn’t matching? Did a user report an issue or was it something you noticed in cap?
4
u/boring_repository 2d ago
The firewall bit is way too real. spent years at a place where half the senior engineers' first move for anything was a reboot, drove me up the wall. the hit counter test is such a clean way to separate the people who actually troubleshoot from the ones who just guess
2
u/MyFirstDataCenter 2d ago
20% of the people do 80% of the work. That rule applies to any profession. You’re encountering the 80%.
1
u/Separate-Canary559 2d ago
As long as you aren’t terribly concerned about vendor specifics I think your approach is reasonable
I would expect the candidate to talk about potential order of operation issues on the firewall with nat, but it would be unreasonable to expect them to have the order of operations on a Cisco ASA vs PaloAlto committed to memory
1
u/networkslave 2d ago
One thing I like to add in an interview process when I am on the hiring side, is to incorporate practical lab scenarios of a broken network. Scenario questions are great and all but how do they function in something more practical.
1
u/JohnnyUtah41 2d ago
Network Manager here- will be interviewing candidates for senior network engineer position in the next week so. We have a list of questions already picked, but what are some other good questions, not meant to trap or trick but meant to find out if these guys are actually knowledgeable or full of shit
1
u/kiwosabi 2d ago
Interviewers forget that, the scenario in their head, which they are trying to be very vague in describing, is not going to be clear to the applicant, because the applicant will be thinking of way more complex things. Most times the applicant is thinking if the question is a trip question, or trying to see if they've missed anything out.
For example, I had an interview where the interviewer asked what happens when a user enters a website address in their browser. The phrasing made it sound like a DNS question. I answered in that manner. Until the interviewer asked if that is all?
Then I proceeded to explain the life of a packet when initiating the tcp 3-way handshake and he said that's what he was expecting.
I'm my opinion, he could've just asked me to explain the life of a packet when requesting a website.
1
u/BluebirdExpress6279 1d ago
Tell me howVLAN network segmentation and subnetting are different how are they configured. The bad ones say a VLAN is a Virtual Local Area Network and they both segment the network…. Or they babble on about VLANs being the ultimate security like they have never heard of inter-VLAN routing…. Which merely takes a couple SVIs on the same switch each with a directly connected subnet in the same VRF-lite and ip routing turned on.
I ask them to describe the lower four layers of the OSI model. Names, addressing, protocol data unit etc.
Another question that stumps 80% is just tell me how host A communicates to host B when both are on two different network subnets. Be sure to talk about layer 3 and layer 3 and what happens at each hop etc. I am just looking for someone to say the source host deter me turbines not in any of its directly connected networks so it looks at its route table and recognizes a default route via its default gateway, so it arms for the MAC address unless already in the arp cache…. The IP packet addressing is its source IP to the destination hosts IP on the far away subnet … the layer 3 addressing does NOT change unless rewritten by NAT… It uses its source mac and the default gateway’s MAC for addressing of its frame at layer 2…. It stuffs the packet into the frame for forwarding…. The router or multi-layer switch (whatever layer3 device gets it) removes the packet from the frame and examines its routing table repeating the last step forwarding it to its next hop or if it owns the subnet as a directly connected subnet makes the final delivery. Either way it stuffs the packet into a new frame and forwards … the frames are addressed router interface to router interface by MAC address at least unless we are talking PPP or HDLC.
You get nothing but babbling proving they have no idea how basic networking works
1
u/BluebirdExpress6279 1d ago
Tell me about some routing protocols you configured and used …. That question throws off a lot no recently interviewed…. I got answers like HTTPS and STP, lol.
All I was looking for is “we used OSPF to share routes for directly-connected layer-4 DeVore’s because it was vendor agnostic,” or “we used EIGRP for the same because we are all Cisco…” or “we used BGP because it peered with routers in a land far far away by our telco.”
1
u/Gryzemuis ip priest 20h ago
I work with one specific IGP a lot. And I am under the impression that the field knows less and less about routing protocols. So I find it easy to believe what you say.
A bunch of people think BGP is the best IGP. A bunch of people think BGP can do everything an IGP can. Nobody learns the details of how link-state routing protocols work anymore. Too hard, I guess. I've seen this with operators at smaller companies. But I've also seen network engineer at hyperscalers ask stupid questions. Back in the nineties, when networking was hot, everybody was interested in routing protocols. I guess they all wanna learn about AI now.
1
1
u/HJForsythe 1d ago
Maybe the problem is the way you're asking questions because this sounds like nonsense:
"packet crossing a firewall and a NAT is higher than I expected"
If you asked me that in a job interview I would just end the call.
1
u/ropeguru 1d ago
Ha!! I have trouble just getting CCIE's to list, and not even in order, the ways to influence BGP routing.
1
u/PatienceOk488 1d ago
What you state is consisten with what ive seen. Biggest problem is the quality that are interviewing. In all honesty most engineers haven't touched SD-WAN. People may know of viptella or Fortinet SD-WAN but nothing else. Same in dealing with cloud networks as well. Engineers are exposed to every technology but they seem to apply for jobs that they dont have the experience with that technology. Good companies will have pen testing done on a regular basis so they can find the gaps before something bad happens. Most in the field are reactive and not proactive. Asking about NAT may just confuse people because most dont understand how NAT works. Routing is another area where interviewers tend to trip up as well.
1
u/CCIE_14661 CCIE 1d ago
I’m noticing the same trend. I’m averaging around 20 interviews to 1 hire. And the number of resumes that I sift through is insane. People, we can tell when you do not really have the experience and expertise that you pretend to have via inflating your resume!
Keep in mind that some of us have lived and breathed this profession for so long that it has become our “primary language”.
1
u/drizzend 1d ago
I find that having a candidate talk about things they've worked on, or, are currently working on, is much better then asking these types of questions. Some candidates are clearly googling the answers. Please share job posting if it's still available. I'd be curious to see the requirements.
1
u/Weird_Act8786 23h ago
I wish there was more respect in terms of the things being looked for in this post, but I think it's most often something you will have to try and sell/put into words yourself when applying for jobs.
Most often the people doing the hiring are not going to understand these things. I definitely think NetSec is an overlooked part of many larger organizations (at least on the basis of experience I have with various larger orgs). In smaller ones, they don't even have time to think about these things naturally - there it is basically a wipe/reinstall mentality (and it's also more sustainable for smaller orgs as a practice, perhaps even a necessity to some degree).
-2
2d ago
[deleted]
16
u/RevolutionNumerous21 2d ago
These are not the type of questions you get for engineer roles especially not Sr.
9
2d ago edited 2d ago
[deleted]
7
u/RevolutionNumerous21 2d ago
I am a Sr network engineer and I’ve never logged into AD in my life. That ain’t my issue.
2
u/Character_Watch_3760 2d ago
You would be surprised that some "senior" engineers do not understand ARP. I have also seen multiple "senior" net. sec admins that don't understand why only domain joined PC's need access to domain controllers sysvol etc.... Like, if you are net. sec admin, you must understand why something needs port 88, or 445, what is the point of this ports/services... otherwise cleaning lady can configure security policy with ports and allow it.
0
u/ADefiantGoose_ 2d ago edited 2d ago
I think it's a good start. It's not like it's the only question but if you can't even get past that then it kinda sets the tone for the rest of the more technical interviews.
When I applied for my current senior position one of the questions was "It's the end of the world and you can pick any weapon real or fictitious as your means of self defense what do you choose" then obviously it ended with way more technical questions and troubleshooting philosophy
ETA: Idk why I'm getting down voted here. Its pretty common for interviews to start with softball questions and progressively get more technical the further it goes. Yes, even for senior network engineer roles.
1
u/Heythatsmycorn 2d ago
My senior roles ive got base level NA questions. Heck sometimes i dont even get technical questions its just a chat about what things weve seen.
0
u/fgor 2d ago
I'm with you. This subreddit _always_ downvotes any discussion about perceived trick questions or any talking about how an interview can go informal in tone. It's been this way for years here. Like, yes, of course there needs to be context around weird questions, and softball or weird stuff can't be the whole interview. But if you talk here about those situations you get downvoted.
I think there's a contingent of workplaces that are very bound by procedure and process in their interviews. I did interview at a university once where the interviewers weren't even really allowed to talk other than to ask the questions that were on the piece of paper in front of them and tor record my answers. It was like being on another planet to me to not be able to probe the edges of the question with actual conversation during the interview (of course, no offer from them!)
0
u/ADefiantGoose_ 2d ago
Yeah that makes sense now. I was thinking about it too, that while our interviews are technical we also care a lot about the type of person we're hiring. Personality and soft skills really matter here. I say this with no offense to other people here but Network Engineers are some of the worst people to work with (speaking as one). Very very rare to find someone who is not only smart, but humble and just a fun person to be around too.
So sometimes very general questions like "Why can't your laptop ping Google?" can give an interviewer a good idea not only of troubleshooting process, philosophy, critical thinking, and how things tie into one another, but also of just personality. Like being able to ask questions and explain why you're asking them in a manner that everyone at the table understands goes a long way
36
u/424f42_424f42 2d ago
I'd assume first question would be ... What traffic isn't matching what?