Magic the Gathering + Monty Python = Nerdiest Thing Ever?
While standing outside a party the other night, basking in the eerie glow of a moon halo, a good friend of mine who has a penchant for skepticism of science turned to me and said, “You realize that science works on an honour system, right? If a scientist makes some stuff up, more often than not, no one’s ever going to find out.” Usually comments like this from him are my cue to stand up and be the champion to the defense of science. But, the thing is, in this instance he’s right. Science does work on an honour system, wherein unscrupulous individuals are free to lie and cheat. Yet, science works. It may not be 100% effective, but despite what some conservative strategists and this hippie friend of mine may have you believe (funny when they agree on something), this honour system actually works pretty well.
“Honour system?”, you say, “what about that gonad busting process called peer review that we always hear scientists moaning about? Isn’t that the check on a scientist’s honour?” Well, no, not at all really; peer review is more of a check on the quality of a paper, not the veracity of it. When an article is submitted to a journal and sent to a reviewer, the reviewers check that the experiments as described are convincing, well-designed, etc. But, they assume that the authors are telling the truth. Not to do so would require something like random spot checks of labs – a ridiculously expensive, totally unfeasible idea. If someone really wants to make shit up in a paper, they can, and there’s really nothing we can do about it. It’s almost like certain components of the mainstream press, where bullshit is spewed forth on a daily basis and never officially corrected.
Yet, as I said, the system works. (I hope that for the NA audience I needn’t belabour this point.) We manage to understand more and more every day, despite the fact that we have no way of knowing for sure if the stuff we read in academic journals is true. But, the question is this then: if science doesn’t have any checks in place, how does it work? Are scientists truly that honourable?
Although I think a good deal of honour does go into it, there is in fact a check in place, it just doesn’t happen at the time of publication. The check on scientists’ honours is whether other labs end up repeating the result in the process of trying to build on it. There are of course problems with this, most notably, an inability to distinguish between published lies and published truths which simply didn’t inspire anyone towards attempting repetition. Nonetheless, when it comes to the false positive rate in science, this process actually does a pretty good job. It is very rare for an unrepeatable result to get incorporated into the scientific community’s discourse for more than a year or two – after a while, scientists get tired of not being able to repeat the result and it gets left in the waste-bins of the memetic landscape.
The fact that this process manages to bring us ever forward in our knowledge does not mean it couldn’t be substantially improved, though. The best thing, in my opinion, would be a change in the culture of science towards publishing papers that show data in all its naked glory. I’m not suggesting that scientists currently hide data as a regular practice (despite what some Lords idiots pranksters would have you believe), but what we certainly all do is try to pick methods for displaying our data that emphasize the point we’re trying to argue. This comes at the expense of making clear what an actual repetition of a result should look like. We dress our data in fancy linens rather than laying it bare for all to see, and thus we don’t really know what each other’s bits look like. (Sorry.)
One example of this is a practice that my PhD supervisor referred to as picking the “typical best example” for a figure. Figures in scientific papers often contain an example image of some raw data which is supposed to be a “typical” example for illustration purposes. These examples help other scientists to know what the effect on the lab bench actually looked like, which is important if they want to try to do it themselves later. The thing is, such images are almost always the best example the authors could find in their data for making their point, rather than the most typical. In and of itself this isn’t so bad – there’s no lying involved and it’s common knowledge that it’s done. Nonetheless, it makes it harder to repeat a result because you don’t know what most of the data on the bench should look like, only what the best data should look like.
Another example is the use of bar plots of the mean with standard error, like the ones to the right of this figure from Drummond & Vowler (2011):
By comparing the actual data (left) with bar plots of the mean with standard error (right), we can see how bar plots might not tell the entire story.
This is, bar none (da-dum-dum, ching), the most ubiquitous form of graphs in scientific papers. The bar shows the average of the data (the mean) and the little black ‘T’ on top of the bar shows an estimate of how much this average would usually change by if we re-did the experiment many times (the standard error). These bar plots of the mean with standard error are a favourite go-to because they are a) clean looking, b) easy to understand, and c) good at illustrating differences between groups. The problem is that both the mean and the standard error have a tendency to mask the actual data. First, if the data has any skew in it, i.e. if it’s a bit lop-sided, then these measurements are completely misleading, as illustrated by the figure. Second, the standard error always decreases with more data points. As such, when there’s a large amount of data the error-bars can look really tight even when the data is very spread out. So, the problem is that if you try to repeat a result which was presented in this sort of bar graph you don’t know what the actual data should be like unless the data was not skewed at all and you use the exact same number of data points. This is, again, a well-known fact in science, so it’s not like anyone’s under any misconceptions. Nonetheless, it would make the entire process of trying to repeat an experiment easier if the data was presented in a manner that didn’t mask anything.
Science progresses largely due to the principle of people trying to replicate each others’ results. This is critical to overcoming the potential hazards of the honour system that we are otherwise in. (Although I think most scientists are totally honest I don’t kid myself that they all are, so checks are important.) Anything we can do to make this process easier is a good thing, I think, and we can start by changing some of the ways that we present our data. The more that we actually show all of the data, as it really is, without too many cosmetic alterations, the better. In summary, scientific papers should be less like lads magazines and more like amateur, hardcore pr0n. (Sorry x2.)
PS – If you want to find more Magic + Monty go here.
by blake
fresh ripostes: