Saturday, November 29, 2008

The Impact Factor hydra

Interesting post on the entrenchment of impact factors in small countries over at A Blog Around the Clock. The third comment down, by Comrade PhysioProf, is particularly troubling, because it's almost certainly true:

The bottom line for impact factor--or whatever other quantitative metric might replace or supplement it--is that people are lazy and time is highly rate limiting in the professional lives of academics. People are always gonna rely on something fast and easy to obtain--like a journal impact factor--than they are on something difficult and time consuming to obtain--like a developed opinion about the solidity and importance of a particular published paper.

Anything that is gonna replace impact factor of journals in which scientists publish papers as a metric for comparative assessment of scientific productivity is gonna have to be as braindead easy to deploy as impact factor.

So we have what you might call the Impact Factor Paradox: getting a real handle on something as slippery as the value of someone's scientific contributions is inevitably going to be time-consuming and hard; metrics that allegedly measure that value are going to fall along a spectrum from "time-consuming but accurate" to "quick, easy, and horribly flawed"; in a system where time is the limiting resource, there will always be a sort of grim undertow toward the quick-'n-greasy metrics.

Your thoughts on how to avoid this trend are welcome.

Labels: ,


Blogger J-Rod said...

I don't have an answer to this. But while reading it, I was reminded of this most recent entry from Paul Graham. These problems are very similar, leading me to believe it's a Human Organization problem, not anything intrinsic to either Academia or the Business World.

The Other Half Of "Artists Ship"

10:09 AM  
Blogger Zachary Miller said...

Hmmmm...good question. How to present a complex subject worthy of further study in a way that "gets to the point" and, based on its brevity, be subject to a misled public?

Publish in Nature or Science!

12:25 PM  

Post a Comment

<< Home