Archive for the ‘management’ Category

Suppose You Get Hit By A Truck   Leave a comment

Sitting here in class (Crisis Management, so far a fun class!), I was struck by an observation that you, general public, may find useful.

Every competent information technology professional I’ve ever met has uttered the phrase, “So what happens when (foo) gets hit by a truck?”  If your IT people don’t ever ask you that question, you may want to look into hiring some new IT people.

Posted January 25, 2011 by padraic2112 in crisis-response, management, work

Audit   1 comment

There are typically four major processes that people talk about when they’re talking about security – identification, authentication, authorization, and audit.  It’s pretty typical for people to talk about the first two as if they were one thing (identification and authorization), but really, they’re not (that’s a topic for another day).

  • Identification: Who are you? – “Are you anybody?”
  • Authentication: Are you allowed to act on behalf of a principal? –  “Are you, the identified person, allowed to play here?”, or “Do we let just anybody play here?”
  • Authorization: What are you allowed to do? – “What sorts of ‘play’ do we allow ‘here’?”
  • Audit: Hey, what have we been letting people do here? –  “Are the above three working?”

I’ll talk about these more in depth someday, but today I want to focus just on audit.

There are lots of different kinds of audit.  You have a computer security audit, whereby some nerd like me analyzes log files and system executables and whatnot and tries to determine if the system itself has only been used for its intended purpose by the people who are supposed to be using it.  You have fiscal audits, where guys in green eye shades analyze accounting logs and purchase orders and credit card receipts and justification forms and try to determine if the money has been used only for its intended purposes by the people who are supposed to be spending it (or collecting it, as the case may be).  You have safety audits, where guys in orange vests with clipboards analyze workspaces and insurance reports and work processes and try to determine if people are doing things that are statistically likely to produce a high number of injuries or deaths.  You have sales audits, where guys in suits look over sales records and market analysis reports and phone logs and try to determine if the guys with good teeth who talk to the customers are selling about what they ought to be expected to sell given the corporate understanding of the market and the customers.

In practice, all these things are wildly different, obviously.  Conceptually, from the standpoint of systems analysis, they’re all the same.  You’re taking some process, and you’re examining the inputs and outputs of that process, and if the end result doesn’t jibe with what you expect, you have a problem.  Either the inputs are off or measured improperly, the process is bad or is measured improperly, the outputs are off or are measured improperly, or your expectation (the way you audit) is just outright wrong.

Now, in the real world, almost everybody *hates* audit.  There’s lots of reasons for this, of course (in many cases, the Big Irk is that the auditor only looks at the first three possibilities, and it’s difficult or impossible to get the auditing organization to see that the actual problem is that they’re doing it wrong).

At the same time, in the real world, everybody *loves* audit, as long as what’s being audited is something somebody else is doing.  Politicians talk about oversight (which is a nice code word for audit), and the public eats it up.

Oversight!  That’s gotta be good, right?

Welfare scofflaws, corrupt politicians, police abusing authority, people abusing government grants, yeah!  Catch those rich bastards putting their money in the Swiss banks and tax the hell out of them!  Crawl up BP’s hind end with a flashlight and find out who’s responsible for this big oil spill!  We want accountability!  Measure teacher performance!  Who’s paying for my congressperson’s reelection campaign!?  Who’s driving, have they passed the test?  Who’s in the country, are they a citizen?  Who’s using welfare that shouldn’t be?  What government programs aren’t producing results?  What the hell are we spending all this money for in the military budget?  Oh, and hey, are our fraud reporting mechanisms actually working at all?  We need to audit our ability to audit!  Rargh!  Righteous indignation!

Somebody knocks on your door and says you’re being audited, suddenly you might not be such a fan of oversight.

Regulations!  Compliance!  Paperwork!  I gotta stand in line at the County Records Office or the DMV!  I have to write a stupid five page report justifying buying a plane ticket on Lufthansa instead of United, what a waste of my frickin’ time!  How the hell am I supposed to be getting any work done with all this bureaucratic red tape getting in my way!  Government is so inefficient!  We can’t measure teachers by performance, it doesn’t work!

Okay, take a breath.

Here’s the reality.  You can audit a process for success, or failure, or both.  Which one you *ought* to use in a particular scenario actually depends upon a wide number of factors.

  • What’s our false positive rate? – how often will our audit flag somebody as being bad, when they’re not?
  • What’s our false negative rate? – how often will our audit flag somebody as being good, when they’re bad?
  • How much does it cost for us to audit this thing, whatever it is?
  • What are the externalities involved in the audit?  Are we auditing the right process to begin with?
  • What happens if we don’t audit anything at all? – does it even matter?
  • If we don’t audit, will the negative consequences actually cost more than the audit?
  • If we do audit, can we do anything with the results, or are we already limited to doing one thing anyway (e.g., “Too big to fail”)?
  • Quis custodiet ipsos custodes?
  • If we make it harder for people to do bad stuff, does this actually prevent people from doing the bad stuff, or does it just make it more profitable for those who get away with it?
  • If that last is the case, are we actually going to have less bad stuff (in toto), or just fewer incidents of bad stuff with a lot more bad in the stuff?
  • If we make it harder for people to do one kind of bad stuff, are they going to stop doing bad stuff altogether?  Or are they going to move to a different kind of bad stuff that’s worse?

These are all questions you need to ask yourself when people start talking about “accountability” and “oversight”.

Otherwise, what you’re paying for isn’t better or more secure processes.  What you’re paying for is a false sense that you’re getting what you’re paying for, which is double-dipped stupidity.

Posted June 8, 2010 by padraic2112 in management, politics, security

Just Don’t Forget the Middle Part   Leave a comment

“Pay them enough to take money off the table” (also remember that “taking money off the table” means something entirely different to a 26 year-old bachelor and a 32 year-old father of two, and that your pay increase method needs to account for that, or it ain’t gonna work).

That reminder aside, an awesome video.  Check out this for more.

Tip o’ the blogger hat to I can’t believe it’s not a democracy!

Posted June 2, 2010 by padraic2112 in economics, management, science

Dr. Rob   1 comment

Dr. Rob over at Distractible Mind has unwittingly written a more universal post than he may have thought he was.

Go read it.  Then copy it into a word processor, and do a search/replace on the word “doctor” with “IT person”.  Or “car mechanic”.  Or “plumber”.  Or “contractor”.  Then replace “patient” with “customer”.

You don’t need to tweak it very much more.  Nice post, DR.  Doctors aren’t the same as other service personnel, of course, for obvious reasons… but in the general sense, people don’t want to hire an expert in any field only to have the expert treat them like a set of boxes on a checklist.

Posted April 1, 2010 by padraic2112 in management

Useful Satire   3 comments

From Coderoom, a post entitled:

3 Simple Rules That Will Make You a ‘Superstar’ Developer

Read the whole thing, if you run any sort of technical or programming project team.  From the end of the post:

Postscript for the naive: This post is a mild satire on programming in teams. These three rules, while undoubtedly effective, are evil. They harm overall project progress for your own benefit. They don’t make you a better programmer intrinsically, only compared to the rest of your team. You may, like I and countless others, have done something like this completely innocently in the past, when you didn’t know better. Now you know better.

Postscript for project managers: If your environment meets the grounds for the Two Fundamental Principles, then you will get programmers playing The Game and your project will suffer. Change the rules. Make sure that programmers are recognised for playing nicely with each other’s code, for working in small teams on larger problems. That rewriting for the sake of it is frowned upon, or that the bugs it introduces are traced back to the rewrite that caused them. I don’t know what the right way to get away from this is. If you do then please, for the love of all projects everywhere, leave a comment!

Posted January 28, 2010 by padraic2112 in management, tech, Uncategorized

An Observation about MBAs   5 comments

In the course of my MSIS/PhD education, I’ve been exposed to a lot of organizational science (or management theory, depending upon how you want to classify the particular chunk of knowledge).  I’ve read some of the seminal works that MBA students read, spent some time perusing the Harvard Business Review and other publications, et cetera.

In many cases, I’ve said to myself, “Yep, that’s true” or “Darn tootin’, that would be the way to go” or some other expression of agreement with what I’ve read.

So here’s the curious part: everywhere I’ve ever worked, there’s been MBA people around.  Everywhere I’ve ever worked, there’s been people who have taken classes in project management, or people management, or both.  These people presumably have read the same material I’ve been reading (one would think).

So why is it so rare to find anyone who actually practices any of it?

It’s a national joke that MBA-types tend to be pointy-haired bosses, right?  It’s the entire premise of Dilbert… people who study MBA material turn out to be terrible at management.

This *can’t* be the fault of MBA programs in general.  There certainly are bad MBA programs, I imagine… but even if you teach this material badly the material itself still seems to be worthwhile reading.

Is it just the case that most people who go and *get* their MBAs aren’t cut out to be managers in the first place?  That there is an underlying set of characteristics of most MBA-seekers that makes them bad at the job they’re ultimately seeking?  I take it for granted that many people who get MBAs are trying to up their chances at higher-paying jobs, of course, but it can’t be the case that self-interest is that tightly coupled with idiocy.

Maybe MBA programs need to focus more on cutting out the chaff?  Harder grading?  Better gatekeeping?  Management is hard, why is it so easy to get an MBA?  Do most programs focus too much on, “learn the material” and not enough on “show you can implement it in practice”?

Posted January 19, 2010 by padraic2112 in management

How To Hire A Sysadmin, Part II   4 comments

As promised, albeit a bit later than originally scheduled, the second part of “How To Hire A Sysadmin”, with the Second Question to ask a potential hire:

“You are sitting at your desk one day when the Chief Operations Officer shows up out of nowhere and says, ‘I believe that my assistant Frank has been communicating company secrets to competitors.  We let Frank go this morning.  I want a copy of all Frank’s files and his email put into the Operations share on the file server so that the Ops group can go through it to see what the damage is.’  What do you say?”

I’ll give you a hint, the best answer is a very quick and unequivocal “I’m sorry, Dave, I can’t do that.”  The second (almost as acceptable) reply is, “I’m afraid you’ll have to have that okayed by the office of legal counsel.”  [edited to add]  This actually can go either way in terms of what’s the “best” answer.  I’m getting ahead of myself.

I’ll give you another hint, hiring someone who answers the question correctly may turn out to be a pain in your butt later.  It’s for your own good.

In spite of the fact that (at least in the State of California) your employees’ email accounts are regarded as property of the company, the fact still remains that you (in this case The Company or The Organization or whatever) have legal responsibilities when it comes to employee data.  From Harvard Law (just one example):

The general idea has been that the employer owns the equipment, and can therefore set the terms of its use. Even under current law, which has been deferential to employer monitoring, this does not mean that employers are free to monitor or not monitor at will. It is not clear, for example, whether employers who fail to notify their employees that they monitor their mouseclicks will avoid liability for invasion of privacy. Moreover, even if employers issue a general notice to employees that they may be monitored, an employee might argue that more specific notice is required.

Even if your workplace has very well defined rules regarding employee’s use of electronic equipment, be aware that you can have all kids of serious repercussions for mishandling data.  If Frank’s mother recently emailed Frank that she had Hepatitis C, and that information winds up getting out, you can be up for a world of painola.

Handling data discovery, even internal to the company, is not something that a systems administrator should *ever* regard as his (or her) blanket responsibility.  They shouldn’t be monitoring general computer usage either, but that’s a subject for another post.  Oh, wait, I already wrote that one.

In the United States, there are three classical professions, namely… doctor, lawyer, and priest according to Wikipedia.  It’s generally given now that this list is somewhat longer, the Department of Labor has a whole list of “professional and related occupations“.

Generally, to be officially regarded as a “profession” (as opposed to an “occupation”) you need to meet a few requirements, one of which is some sort of professional body.  Borrowing straight from the Wikipedia page:

Professions are typically regulated by statute, with the responsibilities of enforcement delegated to respective professional bodies, whose function is to define, promote, oversee, support and regulate the affairs of its members. These bodies are responsible for the licensure of professionals, and may additionally set examinations of competence and enforce adherence to an ethical code of practice. However, they all require that the individual hold at least a first professional degree before licensure. There may be several such bodies for one profession in a single country, an example being the ten accountancy bodies (ACCA, ICAEW, ICAI, ICAS, CIMA, CIPFA, AAPA, CIMA, IFA, CPA) of the United Kingdom, all of which have been given a Royal Charter although not necessarily considered to hold equivalent-level qualifications.

Typically, individuals are required by law to be qualified by a local professional body before they are permitted to practice in that profession. However, in some countries, individuals may not be required by law to be qualified by such a professional body in order to practice, as is the case for accountancy in the United Kingdom (except for auditing and insolvency work which legally require qualification by a professional body). In such cases, qualification by the professional bodies is effectively still considered a prerequisite to practice as most employers and clients stipulate that the individual hold such qualifications before hiring their services.

There is no nationally-recognized “systems administrator” professional body (there ought to be), and yet we have administrative or root access to the file servers upon which sit not just your corporate data, financials, customer lists, etc., but also *your* email and *your* documents (that’s one major reason why there ought to be).  If you are a doctor, a lawyer, an engineer, or an accountant you may actually *lose* the ability to do your job in perpetuity if you fail to exercise your professional responsibilities in the eyes of your professional body; you can lose your license to practice law, or medicine, or submit plans to City Hall, etc.  This is a double-edged sword: if you mess up, you can literally be ejected from the profession.  However, with it comes a protection that currently systems administrators don’t have: if The Boss tells you to do something that violates your professional ethics, you can tell The Boss, “I’m sorry Dave, I can’t do that, because I could lose my license.”  Right now, sysadmins have the responsibility to protect the data, but we don’t have much in the way of business clout.  You want to hire someone who will actively refuse to let you shoot yourself in the foot, even without that backing.

Trust me.

Posted October 27, 2009 by padraic2112 in management, tech

How To Hire A Sysadmin, Part I   5 comments

There’s lots of lists out there for “interview questions” to ask IT people when you are interviewing them for a new position.  Many of those lists are pretty worthless in practice, as they actually ask the sorts of questions to which you can find the answers with 60 seconds and a web browser, but they don’t ask the sort of questions that actually tell you anything about the candidate’s capability to understand complex system design.

I really don’t need to know if you’ve memorized the IPv4 header (this is the networking equivalent of memorizing Pi to 40 digits).  I don’t really need to know if you know the difference between the HKEY_CURRENT_CONFIG and HKEY_LOCAL_MACHINE registry hives on a Windows machine, or what the difference is between GRUB and LILO, or what your opinion is of the advantage of the FreeBSD ports collection vs. Linux’s RPMs.  I *really* don’t need to see your Perl coding skills, because if you’re a really good Perl coder you should be writing code, not administering systems.  Not to mention the fact that if you’re administering a lot of systems with home-grown Perl code I probably don’t want to hire you because after 6 months the only person who will have a freaking clue about how the cluster works is the guy who wrote all the tools from scratch in Perl.

What I need to know is if you understand, at a meta-level, what a sysadmin is supposed to do.  You can learn syntax over time (or ask the magic Internet machine).  Learning how to juggle interdependencies is something else.  In fact, quite often those people who are really skilled at syntax (read: recent certification acquisitions) can sound like they really know what they’re doing, without knowing anything about what they *ought* to be doing.

So, I have only two questions for a sysadmin candidate.  Here’s the first one:

“You have a cluster of 300 machines, running 40 different services, on three discrete networks, with two OS-level dependencies. Assuming you’ve built this cluster yourself from scratch with no legacy dependencies, describe this cluster. Feel free to ask as many questions as you like for clarification. Go.”

This is meta-level information mining.  A good sysadmin will spend more time asking questions about what the cluster is supposed to be doing, what sort of services are running, what’s the uptime requirements, who the users are, and what the business continuity requirements look like than they will talking about their design ideas.  A good sysadmin will have a thousand questions.  Note, you have to be able to provide at least theoretical answers to these questions in order to interview a candidate this way.  Second note, if you can’t interview someone this way, you probably should not be involved in the decision making process for new IT hires.

A *really* good sysadmin will ask questions about the physical facility, budgeting, and office politics, not just technology.  They’re going to want to know if they’re going to be able to fix things based upon technological merit, or if there’s a labrynthine approval process that goes through someone who has no technical expertise but absolute veto power over technology decisions… but if you get someone like this in an interview, be forewarned that you’re either hiring someone who will replace your IT manager within 6 months, or someone who will need some other sort of upward mobility within 18 months or they’re going to get bored and go elsewhere.  The price of hiring really great people is that you need to give them really high level work.

We’ll talk about question #2 in the next post.

Posted October 19, 2009 by padraic2112 in management, tech

Niel Nickolaisen is My Hero   Leave a comment

I have a pile of “CIO Decisions” on my desk.  They’re good bathroom reading for IT people.  The content is not what I would call deep and rigorous, but even when it is just complete fluff it is interesting to see what sort of complete fluff other IT people are currently thinking about.

The September 2007 issue had a final article entitled “How I Fixed My Telco Billing Problems”.  Full article is available online here (you need to register to read it).  From the article:

What am I mad about? Invoices from telecommunications providers (voice, data, cellular). It seems to be standard industry practice for their invoices to be wrong. Not marginally wrong, but very wrong.

No kidding!  When I ran the phone switch and took care of the telco billing at Idealab not a month went by that I didn’t find something wrong, somewhere.  For example, a vendor who shall remain nameless simply forgot that they had provided us with a DS3.  Of course, it was several orders of magnitude more likely that they’d charge us for something (or charge us at some rate) that was incorrect.  Slogging through those bills was a huge time sink.  Niel’s solution?

I am changing the relationship (and my contracts) with my telco providers. A few weeks ago I started the process with one of my major providers. Instead of having someone on my staff scrutinize their bills to ensure they are accurate, I told this provider that it is their responsibility to send me an accurate invoice. If this requires them to hire additional staff to replicate what my staff is doing (or even to hire one of the cottage industry vendors), so be it. Rather than scrutinize their bills for accuracy, I will do some sampling. The first time we find an error, I require the provider to give me a 10% credit, recalculate the invoice, and try again. If the rebill still contains errors, the provider gives me another 10% credit and tries again. Perhaps this will give the providers a financial incentive to get things right the first time.

Brilliant.  Niel, you deserve a bonus.  If I was stuck running telco again, renegotiating those contracts would be job #1.

Posted February 4, 2009 by padraic2112 in management, networking, tech

Downtime: Amazon S3   Leave a comment

ReadWriteWeb reports an outage over at Amazon S3:

Today’s big news is that Amazon’s S3 online storage service has experienced significant downtime. Allen Stern, who hosts his blog’s images on S3, reported that the downtime lasted 3.5 over 6 hours. Startups that use S3 for their storage, such as SmugMug, have also reported problems. Back in February this same thing happened. At the time RWW feature writer Alex Iskold defended Amazon, in a must-read analysis entitled Reaching for the Sky Through The Compute Clouds. But it does make us ask questions such as: why can’t we get 99% uptime? Or: isn’t this what an SLA is for?

A six hour outage does not represent a violation of 99% uptime.  If you’re looking for 99% uptime, you’re looking at 87 hours 36 minutes of downtime every year.  Six hours of downtime is between four and five nines, folks.  If this is the second 6 hour outage of S3, get ready.  You’re 12 hours down, 75 1/2 hours to go in 2008.  Heck, you should be happy, you’re way ahead of the game!

And, as I’ve pointed out before, you’re not getting enterprise service because you’re not paying for it.

Posted July 21, 2008 by padraic2112 in management, news, tech, web sites