Reason #317 to be depressed: journal impact factors

Everywhere I look, I see a common problem. Someone a long time ago came up with a numerical measure of X, and simply because this number exists bureaucrats and administrators have jumped on it as a convenient measure of Y.

One of the most pernicious is the use of student evaluations to decide almost everything. Student evaluations are a popularity contest, pure and simple. Are the best teachers always the most popular? Think about what factors would make a student give a teacher high marks--light workload, infrequent class meetings, easy tests, big curves, and a please-walk-all-over-my-life office hour policy come to mind. Are those the things we really want to be rewarding?

Another is the use of journal impact factors to decide how hot your research is. If you're not familiar with these, the top journals are rated by how frequently their articles are cited. Usually higher impact factors go with more exclusive journals with higher turnover rates. A lot of museum publications and regional journals aren't listed at all. And people (well, bureaucrats) use these things. A lot. So often the decision of whether to hire or promote someone is based less on, "Is their work any good?" than "What journals have they been published in?"

(One obvious loophole is that a paper can be cited a billion times not because it is good but because it is spectacularly bad.)

So there are big problems with journal impact factors even if they were calculated in an honest, transparent way by a disinterested party. They're not.

Thomson Scientific, the sole arbiter of the impact factor game, is part of The Thomson Corporation, a for-profit organization that is responsible primarily to its shareholders. It has no obligation to be accountable to any of the stakeholders who care most about the impact factor—the authors and readers of scientific research. Although we have not attempted to play this game, we did, because of the value that authors place on it, attempt to understand the rules. During discussions with Thomson Scientific over which article types in PLoS Medicine the company deems as “citable,” it became clear that the process of determining a journal's impact factor is unscientific and arbitrary. After one in-person meeting, a telephone conversation, and a flurry of e-mail exchanges, we came to realize that Thomson Scientific has no explicit process for deciding which articles other than original research articles it deems as citable. We conclude that science is currently rated by a process that is itself unscientific, subjective, and secretive.

During the course of our discussions with Thompson Scientific, PLoS Medicine's potential impact factor—based on the same articles published in the same year—seesawed between as much as 11 (when only research articles are entered into the denominator) to less than 3 (when almost all article types in the magazine section are included, as Thomson Scientific had initially done—wrongly, we argued, when comparing such article types with comparable ones published by other medical journals).

Makes me wanna holler.

Thanks to Mike for the tip.