And so it ends… or not?
It’s a weird feeling for me when I see that an article I’ve covered is being retracted. I find myself wondering whether I should have asked more questions about the quality of the work – which may or may not be fair. But after all, these articles have (presumably) gone through peer-review by experts in the field (for whatever that may be worth) and been seen by many thousands of eyes before I got around to reading them, and it’s not like I’m replicating the experiments myself. But I still remember the gut-twinge I felt during the infamous Woo-Suk Hwang stem cell debacle, just a few months after I’d covered their team’s work in a two-page news feature.
Now it’s happened again – coincidentally, with another Science article. This time the circumstances are a bit weirder, though. To be sure, Ferrer and Golyshin were making a pretty big claim – by assembling chips containing arrays of more than 2500 fluorescently-labeled molecules, they hoped to essentially catalogue the various metabolic reactions being performed by a particular cell or microorganism. Their proof of concept, with the bacterium Pseudomonas putida, was an apparent success, but it wasn’t long before the trouble started.
Just two months later, Science published an editorial ‘Expression of Concern‘ from Bruce Alberts, followed by a blog post cataloging some of the criticism. This latter piece presents some serious charges, most notably from the prominent UWisconsin biochemist Laura Kiessling, regarding the group’s ability to successfully assemble and obtain useful results from such chips based on the methods described in their article. And this may be fair; I’m no chemist, but the commenters at Derek Lowe’s In the Pipeline – my go-to site for chemistry and drug discovery info – essentially tore the paper a new asshole. Some of the points they critique (e.g., flaws in the chemical structures) may just be sloppiness on the part of the authors in their haste to get their supplementary info out there with all due alacrity, but there are certainly challenges to the actual chemistry that sound fairly serious. For me, though, the really damning thing to me is this somewhat off-hand remark from the Science blog post (also pointed out by Derek):
The journal’s executive editor, Monica Bradford, acknowledged that none of the paper’s primary reviewers was a synthetic organic chemist.
Excuse me? I’m sympathetic to many/most of the complaints from the peer-review reform movement about the Trouble With Journals Today, but I also spent a number of years in the scientific publishing biz and I generally credit editors with at least trying to pick field-appropriate reviewers. So this is baffling, and it undercuts any ability to make a clear judgment on the paper’s quality based on what we know so far.
But so anyway. At the end of July, Ferrer’s institution, Madrid’s Institute of Catalysis and Petroleochemistry, decided that enough was enough and recommended the paper be retracted, and is now considering whether any heftier penalties are called for. Game over, right?
Not really. As mentioned in the above-linked Nature news report, some of the scientists who reviewed the case think this may have been simply a case of sloppiness:
Some scientists are still convinced that the methodology used to create the array could work. “We don’t know,” says molecular geneticist Pere Puigdomènech of the CSIC’s centre for research and development in Barcelona, who headed the CSIC ethics committee. “We only criticize how the science in this paper was conducted and reported — we’d be very happy if someone could validate it.”
And then today, The Scientist published an interview with Sir Richard Roberts – co-Nobel laureate with Philip Sharp for the discovery of RNA splicing, and current head of enzyme-providers extraordinaire New England Biolabs. Roberts unreservedly defends Ferrer and Golyshin:
I made an arrangement to visit Dr. Ferrer in Madrid during a conference that I was at last December. I spent about three hours or so with him, met with all the people in his group, [who] showed me how they did everything, [and even] went over and met with one of his local collaborators, who was doing some of the identification work. And everything that I saw while I was there gave me complete confidence that everything was above board…
I said, ‘Why don’t we just set up a test?’ So that’s what we did. Basically, we set up 10 compounds, of which 8 [Ferrer] should be able to detect and know what they were, one should not have been detectable at all by his system, and the other one was a complete unknown. And the bottom line was that he got the correct results on everything.
So where does that leave us? At this rate, I imagine that a retraction may already be imminent. But it sounds to me as if a lot of the complaints are based purely on skepticism and the sloppiness of both the authors and the journal, rather than on direct demonstrations that the technique itself is unworkable. The criticisms are withering and fairly convincing, and come from extremely reputable corners, but as far as I can tell, they remain grounded on a review of what seems by all accounts to be a very poorly written methods section rather than experimental efforts to demonstrate workability.
And as far as I’ve seen, Roberts is the only scientist who has replicated the reactome method – and he claims to have succeeded unequivocally. Unfortunately, that hardly vindicates the method – and I imagine there will be a great many more tests before we see reactome chips popping up again… in the meantime, it looks like we’ll have to continue to comb through the growing mounds of genomic data the hard way.