There’s lots of lists out there for “interview questions” to ask IT people when you are interviewing them for a new position. Many of those lists are pretty worthless in practice, as they actually ask the sorts of questions to which you can find the answers with 60 seconds and a web browser, but they don’t ask the sort of questions that actually tell you anything about the candidate’s capability to understand complex system design.
I really don’t need to know if you’ve memorized the IPv4 header (this is the networking equivalent of memorizing Pi to 40 digits). I don’t really need to know if you know the difference between the HKEY_CURRENT_CONFIG and HKEY_LOCAL_MACHINE registry hives on a Windows machine, or what the difference is between GRUB and LILO, or what your opinion is of the advantage of the FreeBSD ports collection vs. Linux’s RPMs. I *really* don’t need to see your Perl coding skills, because if you’re a really good Perl coder you should be writing code, not administering systems. Not to mention the fact that if you’re administering a lot of systems with home-grown Perl code I probably don’t want to hire you because after 6 months the only person who will have a freaking clue about how the cluster works is the guy who wrote all the tools from scratch in Perl.
What I need to know is if you understand, at a meta-level, what a sysadmin is supposed to do. You can learn syntax over time (or ask the magic Internet machine). Learning how to juggle interdependencies is something else. In fact, quite often those people who are really skilled at syntax (read: recent certification acquisitions) can sound like they really know what they’re doing, without knowing anything about what they *ought* to be doing.
So, I have only two questions for a sysadmin candidate. Here’s the first one:
“You have a cluster of 300 machines, running 40 different services, on three discrete networks, with two OS-level dependencies. Assuming you’ve built this cluster yourself from scratch with no legacy dependencies, describe this cluster. Feel free to ask as many questions as you like for clarification. Go.”
This is meta-level information mining. A good sysadmin will spend more time asking questions about what the cluster is supposed to be doing, what sort of services are running, what’s the uptime requirements, who the users are, and what the business continuity requirements look like than they will talking about their design ideas. A good sysadmin will have a thousand questions. Note, you have to be able to provide at least theoretical answers to these questions in order to interview a candidate this way. Second note, if you can’t interview someone this way, you probably should not be involved in the decision making process for new IT hires.
A *really* good sysadmin will ask questions about the physical facility, budgeting, and office politics, not just technology. They’re going to want to know if they’re going to be able to fix things based upon technological merit, or if there’s a labrynthine approval process that goes through someone who has no technical expertise but absolute veto power over technology decisions… but if you get someone like this in an interview, be forewarned that you’re either hiring someone who will replace your IT manager within 6 months, or someone who will need some other sort of upward mobility within 18 months or they’re going to get bored and go elsewhere. The price of hiring really great people is that you need to give them really high level work.
We’ll talk about question #2 in the next post.