The Impact Factor hydra

Interesting post on the entrenchment of impact factors in small countries over at A Blog Around the Clock. The third comment down, by Comrade PhysioProf, is particularly troubling, because it's almost certainly true:

The bottom line for impact factor--or whatever other quantitative metric might replace or supplement it--is that people are lazy and time is highly rate limiting in the professional lives of academics. People are always gonna rely on something fast and easy to obtain--like a journal impact factor--than they are on something difficult and time consuming to obtain--like a developed opinion about the solidity and importance of a particular published paper.

Anything that is gonna replace impact factor of journals in which scientists publish papers as a metric for comparative assessment of scientific productivity is gonna have to be as braindead easy to deploy as impact factor.

So we have what you might call the Impact Factor Paradox: getting a real handle on something as slippery as the value of someone's scientific contributions is inevitably going to be time-consuming and hard; metrics that allegedly measure that value are going to fall along a spectrum from "time-consuming but accurate" to "quick, easy, and horribly flawed"; in a system where time is the limiting resource, there will always be a sort of grim undertow toward the quick-'n-greasy metrics.

Your thoughts on how to avoid this trend are welcome.

Impact factors, copyright law, and other science publishing buzzwords

One of my former labmates just sent around the latest list of journal impact factors. I'd repost 'em here, except that I don't want to perpetuate the perception that they are important (although I was grateful to have seen the last version, in a checkout-stand-tabloid-curiosity sense). Here's why:

People seem prone to forgetting that journals have impact factors because individuals papers are cited widely, or not. It's not like every paper that Nature (IF = 28.something) publishes is widely cited, or every paper that Canadian Journal of Earth Sciences (IF = 0.955) publishes is not. In a field as small as sauropod paleobiology, everyone is going to read all of the literature no matter where it is published (the last remaining exception being obscure foreign journals that are not easily available as PDFs; and I mean foreign here as in "outside any researcher's country", not just "outside the US"; getting hold of un-PDFed papers from Oklahoma Geology Notes is probably a cast iron bitch if you're in a local museum in China). The only real advantage of a high-profile journal is to possibly bring the paper to the attention of non-sauropod workers. Whether Nature does that any better than CJES is probably up for discussion (at least in the small world of sauropod paleobiology). Unless some avian physiologist or human bone biomech person is going to have their world rocked by what I write, that 'crossover' appeal is probably not worth stressing over too much.

And there is an entirely different form of impact that we need to be thinking about, which is: how easy is it for people who are interested in your research area to find out about your work and acquire it? I'll bet that my effective impact is much greater than it would be otherwise simply because all of my papers are freely available online (thank you, Mike!). Although individual researchers are doing this more and more--see the badly-in-need-of-updating list here.

And there's a third kind of impact that is arguably more important than either of other two, which is: how big is the intellectual footprint of a given paper? For example, I'd argue that Kristi Curry-Rogers's paper on Apatosaurus ontogeny in JVP has been far more important and influential than her paper on Rapetosaurus in Nature (feel free to argue otherwise, I'm just shooting from the hip here).

Seems to me that what matters for Impact(3) is quality and timeliness of ideas (and timeliness often trumps quality), and publication venue is almost completely irrelevant (although I'd be interested in seeing a counterargument). What matters for Impact(2) is availability of work, which is better for papers that came out in open access journals but easily remedied for papers that didn't (at least until publishers crack down on free distribution of PDFs, if ever*). And Impact(1) is not unimportant, but it's also not nearly as important as people think it is; indeed, Impact(2) and Impact(3) actively erode the importance of Impact(1).

The main thing propping up Impact(1) as something we all have to at least pretend to care about is that the perceived prestige of having published in a high-IF journal really does matter to bean-counters in universities and funding agencies who can't be bothered to assess the actual quality of someone's work and for whom a nice convenient number is a godsend, even if it is horribly flawed. Therefore it also matters to many of us, who can't get ahead without having the approbation of those faceless but immensely powerful entities. See also: student teaching evaluations.

If we're going to be stuck with one number to evaluate something like the impact of a scientific paper, how about the number of citations that pop up on Google scholar? It's fast, free, and doesn't pretend to be anything other than what it is: the number of papers indexed by Google Scholar that cite the paper in question. Tragically, I suspect that ISI impact factors are popular in bureaucracies specifically because they are slow, inaccurate, and not transparently available to all.

* Recently there was a post on the Vert Paleo listserv about how copyright is a barrier to making the existing backlog of literature easily available electronically. My feeling is that those of us needing to get work done will scan what we need ourselves, circulate it through private channels, and keep plowing on. Those who try to stop us with be publicly humiliated at worst (when they come down on us and we all collectively vow never to use their stuff again) or marginalized at best (when we just stop using their stuff because it's harder to get hold of).

Finally, if you haven't been following the blogosphere brush fire (or brush war?) that sprung up in the wake of Nature's pathetically transparent slam of PLoS, Greg Laden has kindly assembled about a million relevant links here.

Discuss.

More glittering gems from my outbox: finishing the dissertation

I don't update this thing very often, but I'm on e-mail all the time, and the profound wisdom and sparkling wit just comes jetting out of me like one of those lava fountains in Iceland. So I'm going geothermal here.

A recent correspondent asked me about dissertation stuff, and this is part of my response:

-------------------

....The most honest and useful advice I ever heard about dissertations came from Darren. I wrote to ask him what finishing was really like (this was when I was still a few months out). Tragically I've lost his original reply, but the gist was, "Eventually you will realize that you can't finish by treating the dissertation like a day job. It is a godforsaken monster that will continue to ruin your life until finally you've had enough and you just decide to kill it. After that, you may put in impossible hours and push yourself right to your limits, but you will finish. It's the only way I know to do it." [Darren, if you have the original thing you sent, please feel free to resend it or post it as a comment. UPDATE: Darren just posted it in a comment, below, and as I suspected it's much better than my paraphrase.]

That was certainly true in my case. The only thing that finally made me face finishing was my committee's insistence that I needed to get it done that semester. And even then I spent the first half of the spring of 2007 treating the diss. like a day job. It wasn't really until the final month and a half that I went into overdrive, literally staying up past my bedtime every night and gradually letting every other concern in my life slide. It sucked, bigtime. And after I filed I didn't look at the diss. at all for about three months. But I did get it done and filed.

It might be possible to just decide, "Okay, I'm going to finish my dissertation now," no matter where in the process you are, but I doubt it and I wouldn't recommend it even if it is possible. I think it's more like trying to climb Everest when there's a storm coming. There is some point up the side of the mountain when you are close enough to the summit to make a dash for it. There's no point in starting the dash until you're that close; otherwise you'll just exhaust yourself and possibly die of exposure. So for now just keep grinding away, a little here and a little there, until you are (a) sick to death of it and (b) close enough to make a run on the summit.

I read some books on writing when I was trying to write my diss--possibly more displacement activity--but none of them were worth a damn. The book that I wish I'd had then but only discovered recently is Steven Pressfield's The War of Art, which I cannot recommend highly enough. It's shelved in the Self-Help section at the bookstore, which strikes me as perverse. It's not really that. It's more like a personal philosophy on how to think about your work and all the things that keep you from getting it done....

------------------

I do recommend The War of Art, to everyone. Jarrod put me onto it. It's a short read, but pithy, and decidedly non-lame. I put it up there with Paul Graham's essays in terms of great writing about how to do useful stuff in the real world.

Oh, yeah, don't forget to actually DESCRIBE your critters, too

In the previous post I went off on people who perpetuate taxonomic chaos by naming new taxa in unpublished theses. Now we're going on to the next step: actually describing the beasts. A lot of new critters are described in very brief notes, especially if they are sexy enough to get into Nature or Science.* As Jerry Harris pointed out in a comment:

there's a sad tendency for authors to perceive these "preliminary" publications as "definitive," even though they provide virtually no useful details, so nothing longer and more detailed gets published for years and years, if ever

Quite.

Here's a plea, from a paleobiologist who needs things like measurements and detailed morphological descriptions to do his work (say it with me: a list of character codings is NOT a description): if you are sitting on one of these named-but-not-really-described taxa**, and you don't have the time or inclination to write and publish a proper description, please, please farm out the project to someone else. The fact is that rapid-fire papers and phylogenetic analyses rarely have legs, whereas monographic descriptions get cited for decades. Osborn 1899 (Diplodocus), Hatcher 1901 (Diplodocus), Hatcher 1903 (Haplocanthosaurus), and Riggs 1904a and b (Apatosaurus and Brachiosaurus) are all into their second century of healthy citation and they are not likely to expire any time soon. I'm sure that Osborn & Mook 1921 (Camarasaurus) and Gilmore 1925 and 1936 (Camarasaurus and Apatosaurus) will last just as long. Not to mention the monographs of Werner Janensch, which are so badass and so important that they are actually used by English-speakers! (Some translations available here.) So it shouldn't be hard to convince someone to take on the project: "Hey, how would you like to write a paper that people will still be reading, citing, and actually benefiting from long after you're dead?" Of course you'll want the person who actually does the work to put you on as an author, but that seems like a small price to pay to getting handed a project that will make them well-nigh immortal in the published literature.

This sort of thing would make a wonderful Master's thesis for someone, as long as they actually publish it when they're done. Of course I am coming from the position that all masters students who are actually generating knowledge (and if you're in the sciences, you'd better be) should be expected to publish their results, and coerced if necessary, but that's a rant for another day.

Incidentally, Jerry holds the highest moral ground when it comes to this problem: after publishing a short description of his new sauropod, Suuwassea, in 2004 (I say short, but it was still about 20 times meatier than anything you can get away with in Nature and Science), he published follow-up papers on the skull, the axial skeleton, the appendicular skeleton, and the phylogenetic relationships of the beast. And that wasn't a matter of padding his CV by generating LPUs (Least Publishable Units): the original paper and the four follow-ups are all between 14 and 31 pages long. Except for Camarasaurus, which has received thorough treatments from Osborn (twice), Gilmore, and McIntosh (twice, plus a monograph on the skull), I don't know of a more thoroughly described sauropod in the world. And Cam has an unfair lead in time (130 years, compared to 3 for Suuwassea) and number of finds (dozens of skeletons, hundreds of specimens, compared to 1).

So if you're tempted to bitch about having to read five papers on one animal, ask yourself: is anyone bitching because they have too much information available about your own critters? Or is it more likely the opposite?

Remember, kids, Dr. Vector says: aspire to Harrisian completeness, not...er, Copeian (safe because he's dead, although more recent examples rush to mind *cough*) rush jobs.



* It totally, totally sucks that ISI ratings, job search committees, tenure review committees, and departmental review committees all view these glorified abstracts as more important and worthwhile than big monographic descriptions that actually have lasting value. (Yeah, that's it! I haven't published in Nature or Science yet because I'm too good.)



** Here's a little worksheet you can go through to see if this description fits you:

1. Was the animal described in Nature or Science or a JVP Rapid Communication?

2. Did you ever publish anything else on it?

3. If I wanted to know, say, the lengths of each of its vertebrae, or its height at the hip, could I get that information out of what you've published?

4. Are you still asking other people who have seen the specimen not to publish on it almost a decade after the initial "description" came out?

Finishing the dissertation

You're going to find that there is a difference between knowing the path, and walking the path.






I can't recall the taste of food. Nor the sound of water. Or the touch of grass. There's nothing…no veil between me and the wheel of fire.






Do, or do not. There is no try.






That's one small step for man, one giant leap for mankind.






ARE YOU NOT ENTERTAINED? IS THIS NOT WHY YOU ARE HERE?






Great, kid. Don't get cocky!

Science: the poster

We often hear about the unity of the sciences, but it's nice to see it.

From Seed:

This map was constructed by sorting roughly 800,000 published papers into 776 different scientific paradigms (shown as pale circular nodes) based on how often the papers were cited together by authors of other papers. Links (curved black lines) were made between the paradigms that shared papers, then treated as rubber bands, holding similar paradigms nearer one another when a physical simulation forced every paradigm to repel every other; thus the layout derives directly from the data. Larger paradigms have more papers; node proximity and darker links indicate how many papers are shared between two paradigms. Flowing labels list common words unique to each paradigm, large labels general areas of scientific inquiry.

Information Esthetics is giving away free posters. You just pay the shipping. And it's cheaper to order several at once (I'm talkin' to you, Berkeleyites).

Somebody put this on a t-shirt, stat!

IMMEDIATE UPDATE: no t-shirt (yet), but if you like this there is a lot more at this page, where the guy who came up with this explains why and how. In particular, there is a big comparative diagram that compares the science strengths of ten nations (US, UK, Germany, Japan, China, etc.). Very cool stuff.