Okay, So I Bent The Truth   4 comments

This post was supposed to be a follow-up to “How To Hire a Sysadmin”, but I’ve been a little busy studying for a midterm and delving into the capabilities of Alfresco, so I haven’t had a chance to write that post up yet.

In the meantime, this came across my radar from the ISWORLD mailing list and I needed to plunk it somewhere where I wouldn’t forget about it (del.icio.us all too often turning into a pit): Open Knowledge Creation: Improving the Peer Review and Adoption Process.  FTA:

The practice of peer review and acceptance has been in place for many years, predating the Internet, and has recognized shortcomings. The Internet has proven to be a disruptive technology and a means for innovation in many areas of science and society. In this paper we offer an organizing framework aimed at redesigning the peer review and adoption process, referred to as open knowledge creation. The framework proposed utilizes the Internet, Google’s Knol and Groups technology. The open knowledge creation framework consists of four stages: creation, review/revision, evaluation/adoption and publication and is intended to offer journals an alternative for the communication of research that more fully exploits the Internet.

Deserves a thorough read-through and analysis.  Drive-by science bloggers from other fields: what’s your take?

Advertisements

Posted October 26, 2009 by padraic2112 in information science, science

4 responses to “Okay, So I Bent The Truth

Subscribe to comments with RSS.

  1. With respect to the authors, this paper is about eight years too late. Jimmy Wales put Wikipedia up (which while lacking in ‘academic credibility’ and vulnerable to vandal edits, works well as a shared knowledge platform) a while back, and while Knol is a bit more specialized a platform than Wikipedia they have a common heritage. No matter how you slice it, the big problem with publishing science is the same regardless of whether or not you are using journals, Wikis, Knol, web pages, etc. It is this: who controls the content? In journals it is a ‘black box’ of (mostly) anonymous peer reviewers. On Wikipedia it is a ‘black box’ of (mostly) anonymous admins. In the proposed OKC format I notice that the authors dodged explicit discussion of the subject, but I’d bet that there would be a ‘black box’ of (somewhat to mostly) anonymous peer editors and reviewers, because that’s the tradition in science.

    Ok, so yes, Knol is possibly a whole lot better format for information transfer than Wikipedia, but then again, who uses it? Why do we need to have a repository of data that is impenetrable to the masses? Why do we need anonymous peer reviews or unseen editors?

    The problem doesn’t seem to have an easy solution, no matter how you slice it. If anyone can contribute, how do you validate content? If everyone is anonymous, how do you validate contributions and edits? If there is no rating or discussion tool, how do you provide peer review? And who decides what a ‘peer’ is, anyway? If a 5th grader finds a legitimate error in an article in Nature or The New England Journal of Medicine, and yet we discount his input because he isn’t a published author, are we not preventing the advancement of knowledge?

    I think something combining elements of multiple common tools like Digg, Wikipedia, Facebook, Yelp, etc. would be a good candidate. Here’s what I want out of a knowledge transfer vector:

    -I want to know all the elements of a traditional science paper (abstract, intro, methods & materials, data, discussion, conclusion, citations, contributors).
    -I want to anyone who rates, writes, reviews, edits, etc, as well as their ENTIRE history of usage of the tool to be public and NEVER subject to edits or retcon without significant public review.
    -Furthermore, I want citations to (where possible) be hyperlinked and indexed so that if an article uses a citation to reinforce a point, the link takes me directly to the relevant section of the cited article.
    -I want to be able to submit reviews and markups of any and all content, and have all of the markups and comments permanently tied to my online identity.
    -Reviews and content should be specific enough that if you agree with 99% of a paper, but have an issue with a single section, statement, or point of data, that you can explicitly criticize it.
    -Ditto for any other markups.
    -All comments and reviews are public, and can only be removed or changed by the author of the edit or review.
    -I’d love a clean interface for discussion of article content without making the original document too messy, but that would not stifle ongoing commentary.
    -Some sort of text indexing that links to related articles would be good too (as well as potentially eliminating plagiarism entirely).
    -Social networking tools for collaborating would be a must as well.
    -Some sort of online stats review would be slick too, so that submitting authors could vet out their P values in advance.
    -Having a ‘draft’ mode where papers are not subject to public view while in process would be ok, as long as once they were ‘published’ they were no longer so protected.

    Bottom line is this: we may never change the culture in science that stifles innovation, discourse, critical review, and cliquishness, but at the least we can restore something currently absent from printed journals: accountability. In addition, we could eliminate the process of blackballing certain papers or authors. If a paper is crap, *I* want to decide it’s crap. I’d prefer not to have some editor at Science do it for me.

    That’s the Scientific Method, baby. 🙂

  2. > If anyone can contribute, how do you validate content?

    You can only do this over time, and it only works if everyone agrees on what is “valid”.

    > If everyone is anonymous, how do you validate contributions and edits?

    That’s less difficult, but still not easy.

    > -I want to anyone who rates, writes, reviews, edits, etc,
    > as well as their ENTIRE history of usage of the tool to
    > be public and NEVER subject to edits or retcon without
    > significant public review.

    Yeah, this is the tough one, most of the rest of what you want isn’t terribly difficult. This requires an authoritative identification, one that cannot be revoked or discarded by the individual (if Professor Jones want to rate, write, review, edit, etc., he (or she) has to do it as “Professor Jones: ID fingerprint FOO1294923586”, and nobody else). That’s hard, you have to have some sort of authoritative identification mechanism. You can’t do public/private keypair, unless you’re establishing nodes of trust, like PGP… and then you don’t have quite what you’re looking for. It’s possibly better than the alternative, which is basically the Ministry of Truth, however.

    • It’s possibly better than the alternative, which is basically the Ministry of Truth, however.

      But isn’t the alternative the status-quo quagmire of patents, corporate ownership of “intellectual property”, and lawsuits aplenty? I would have thought the Ministry of Truth would have a better stranglehold on its cash cow than it appears to have presently.

      • No, sorry, I wasn’t clear… I mean, the alternative alternative to the status quo. We don’t have the Ministry of Truth now. Hm, okay, I have another post to write…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: