EGRU-blog

As a spectator sport, science is usually pretty boring.  But occasionally something comes up that makes you think it’s worth printing tickets, starting a popcorn franchise to cater for the punters craving the spilling of academic blood.  This one’s so good I had to disperse quotes from it through the post.

Perhaps the appeal of what NCPA claims to accomplish is simply too much of an allure for empiricists to abandon the approach, especially because there is no single methodological substitute

The occasion for this sport is the savaging of an idea called “nested (phylogenetic) clade analysis” (NPCA). This was developed in the 1990s to distinguish between different biogeographic hypotheses (such as range expansions, fragmentation etc.), based on the structure of haplotype trees (which map the similarity of genotypes) and the distance between samples.

A haplotype network, yesterday. Source

It sounds like a great idea, but does it work?

There is not a single analysis, simulation or otherwise, that has shown that NCPA can accomplish what it is purported to do—infer multiple historical processes that may characterize a species’ history.

Well, apparently not so well. This is the argument of a commentary in the latest uissue of Evolution by L. Lacey Knowles. Knowles’ comments stem from a recent paper in Molecular Ecology, which examined the performance of NPCA by simulating a simple biogeographic history, and seeing how often NPCA inferred the correct history. And they found that, just as in a previous study, it did this in a massive 25% of simulations.

The simulation studies both clearly show that NCPA incorrectly identifies significant geographic associations at a disturbingly high rate, which leads to inferences about process that never occurred.

The response of Templeton was to point out that with real data NCPA correctly inferred the process that was a priori expected to occur 88% and 62% of the time. But of the times it didn’t do this, it tended to infer isolation by distance, which is what it incorrectly inferred in the simulated data. So, it looks biased. What, then is one to do when one doesn’t know the real process?

However, if NCPA cannot correctly infer a population history for a simple evolutionary scenario, why should anyone have faith in its ability to infer a complicated evolutionary history (Knowles and Maddison 2002)? This is a straightforward and legitimate question that has yet to be answered. It is not complicated—there is no need to obfuscate the issue.

Knowles is pointing out that we have a method that has been extensively used (over 1700 citations), but whose behaviour has hardly been explored. Elsewhere this week I bemoaned the production of meaningless statistics. NPCA goes one step further: it’s a whole algorithm to produce meaningless results. I can see the appeal of this: we would like to have a Wonder Method that we can just feed out data into, and it will spit out some analysis that tells us exactly what is going on. But most of these methods don’t live up to the hype – not even if you use a prior and call them Bayesian. There have to be based on some sort of model, and that will be wrong, to a greater or lesser extent. A black box is going to have a hard time working out how the model is wrong, and whether this matters. That requires some understanding of the data and the context it was collected in (e.g. if you only have 4 populations, whether there are unsampled populations between them).

As noted above, the conditions when NCPA is likely to provide reliable results have not yet been demonstrated. It is therefore difficult to understand the rational for using an unreliable method for generating plausible hypotheses, let alone as the sole basis for historical inference.

Will this be the death knell of NPCA? I suspect not. The allure of what it promises to do will be too strong. What will happen is that biologists will use the technique, and in their manuscripts reference the papers that say it doesn’t work, just to show that they have read the literature. Yes, honestly Dr. Referee, I have. This doesn’t mean they have actually read and understood the message. I know this from experience – a few years ago I wrote a paper saying that species richness are useless, but there is still a regular stream of papers examining their performance. Some of them even cite me.

So, in practice NPCA isn’t dead, even if it should be. The best we can hope for is that someone does a lot of work to see if it can be resurrected – perhaps it needs to be recalibrated to get the right results. It would be helpful, though, if it was calibrated properly, with data from a wide range of known processes, so that it does better than getting it wrong 75% of the time.

I’ll end with my favourite line:

I suppose you might find some solace in error rates of 38% and 23%, after all they are indeed lower than an error rate of 75%.

Reference

L. Lacey Knowles, L.L. (2008). Why does a method that fails continue to be used? Evolution 62, 2713 – 2717. DOI: 10.1111/j.1558-5646.2008.00481.x