Audit   1 comment

There are typically four major processes that people talk about when they’re talking about security – identification, authentication, authorization, and audit.  It’s pretty typical for people to talk about the first two as if they were one thing (identification and authorization), but really, they’re not (that’s a topic for another day).

  • Identification: Who are you? – “Are you anybody?”
  • Authentication: Are you allowed to act on behalf of a principal? –  “Are you, the identified person, allowed to play here?”, or “Do we let just anybody play here?”
  • Authorization: What are you allowed to do? – “What sorts of ‘play’ do we allow ‘here’?”
  • Audit: Hey, what have we been letting people do here? –  “Are the above three working?”

I’ll talk about these more in depth someday, but today I want to focus just on audit.

There are lots of different kinds of audit.  You have a computer security audit, whereby some nerd like me analyzes log files and system executables and whatnot and tries to determine if the system itself has only been used for its intended purpose by the people who are supposed to be using it.  You have fiscal audits, where guys in green eye shades analyze accounting logs and purchase orders and credit card receipts and justification forms and try to determine if the money has been used only for its intended purposes by the people who are supposed to be spending it (or collecting it, as the case may be).  You have safety audits, where guys in orange vests with clipboards analyze workspaces and insurance reports and work processes and try to determine if people are doing things that are statistically likely to produce a high number of injuries or deaths.  You have sales audits, where guys in suits look over sales records and market analysis reports and phone logs and try to determine if the guys with good teeth who talk to the customers are selling about what they ought to be expected to sell given the corporate understanding of the market and the customers.

In practice, all these things are wildly different, obviously.  Conceptually, from the standpoint of systems analysis, they’re all the same.  You’re taking some process, and you’re examining the inputs and outputs of that process, and if the end result doesn’t jibe with what you expect, you have a problem.  Either the inputs are off or measured improperly, the process is bad or is measured improperly, the outputs are off or are measured improperly, or your expectation (the way you audit) is just outright wrong.

Now, in the real world, almost everybody *hates* audit.  There’s lots of reasons for this, of course (in many cases, the Big Irk is that the auditor only looks at the first three possibilities, and it’s difficult or impossible to get the auditing organization to see that the actual problem is that they’re doing it wrong).

At the same time, in the real world, everybody *loves* audit, as long as what’s being audited is something somebody else is doing.  Politicians talk about oversight (which is a nice code word for audit), and the public eats it up.

Oversight!  That’s gotta be good, right?

Welfare scofflaws, corrupt politicians, police abusing authority, people abusing government grants, yeah!  Catch those rich bastards putting their money in the Swiss banks and tax the hell out of them!  Crawl up BP’s hind end with a flashlight and find out who’s responsible for this big oil spill!  We want accountability!  Measure teacher performance!  Who’s paying for my congressperson’s reelection campaign!?  Who’s driving, have they passed the test?  Who’s in the country, are they a citizen?  Who’s using welfare that shouldn’t be?  What government programs aren’t producing results?  What the hell are we spending all this money for in the military budget?  Oh, and hey, are our fraud reporting mechanisms actually working at all?  We need to audit our ability to audit!  Rargh!  Righteous indignation!

Somebody knocks on your door and says you’re being audited, suddenly you might not be such a fan of oversight.

Regulations!  Compliance!  Paperwork!  I gotta stand in line at the County Records Office or the DMV!  I have to write a stupid five page report justifying buying a plane ticket on Lufthansa instead of United, what a waste of my frickin’ time!  How the hell am I supposed to be getting any work done with all this bureaucratic red tape getting in my way!  Government is so inefficient!  We can’t measure teachers by performance, it doesn’t work!

Okay, take a breath.

Here’s the reality.  You can audit a process for success, or failure, or both.  Which one you *ought* to use in a particular scenario actually depends upon a wide number of factors.

  • What’s our false positive rate? – how often will our audit flag somebody as being bad, when they’re not?
  • What’s our false negative rate? – how often will our audit flag somebody as being good, when they’re bad?
  • How much does it cost for us to audit this thing, whatever it is?
  • What are the externalities involved in the audit?  Are we auditing the right process to begin with?
  • What happens if we don’t audit anything at all? – does it even matter?
  • If we don’t audit, will the negative consequences actually cost more than the audit?
  • If we do audit, can we do anything with the results, or are we already limited to doing one thing anyway (e.g., “Too big to fail”)?
  • Quis custodiet ipsos custodes?
  • If we make it harder for people to do bad stuff, does this actually prevent people from doing the bad stuff, or does it just make it more profitable for those who get away with it?
  • If that last is the case, are we actually going to have less bad stuff (in toto), or just fewer incidents of bad stuff with a lot more bad in the stuff?
  • If we make it harder for people to do one kind of bad stuff, are they going to stop doing bad stuff altogether?  Or are they going to move to a different kind of bad stuff that’s worse?

These are all questions you need to ask yourself when people start talking about “accountability” and “oversight”.

Otherwise, what you’re paying for isn’t better or more secure processes.  What you’re paying for is a false sense that you’re getting what you’re paying for, which is double-dipped stupidity.

Advertisements

Posted June 8, 2010 by padraic2112 in management, politics, security

One response to “Audit

Subscribe to comments with RSS.

  1. Pingback: There’s A Flaw In Your Logic, Good Sir « Pat’s Daily Grind

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: