I read an intriguing and slightly baffling quote by Ernest Rutherford the other day; apparently, he once said: “If your experiment needs statistics, you ought to have done a better experiment.” I guess in his day the fields of psychology/psychiatry weren’t considered scientific. Statistics are so much at the centre of what I do, they’re really the sharpest tools in my PhD toolbox. However, statistics without knowledge, conceptual understanding and a thoughtful study design are worse than useless, potentially leading to all sorts of trouble like the dreaded Type 1 error (i.e. a false positive finding), which can easily occur if you simply run enough tests – see here for a good illustration of this. So in effect, statistics are a vital set of tools (at least in psychology/psychiatry – perhaps physicists will have some insight into what Ernest Rutherford was talking about) which need to be used in the right environment and in an appropriate way.
Having come a long way since learning about t-tests in my first year as an undergrad, it seems that hardly a day passes without me finding out something new about statistics. The funny thing is that I don’t really especially go out of my way to do so. It’s just that the more I look at real data sets full of real numbers describing real people, the clearer it becomes that those “textbook” examples you first see when you come across a new stats technique, can be fairly unhelpful. The problem is that so many methods rely on a number of important assumptions, like the residuals of the model you run being distributed at random (i.e. normally around the mean, like a bell curve) or variables not being overly related to one another in a given test. If these assumptions are violated, you need to backtrack and take another route. In the last couple of months, I’ve seen massively zero-inflated distributions, bi-modal distributions, heavily skewed distributions and not-positive-definite matrices. It gets to the point where seeing a normal distribution gives me a warm and fuzzy feeling inside.
In trying to learn how to correctly tackle data with these quirks, I occasionally get the impression that some other researchers either don’t notice these problems or perhaps ignore them. A bit like the impression I had when I discussed the problem of missing data previously. Though, to be fair, some of the techniques of how to address such problems and discussions around how best to use them are relatively new. Another problem is when there just aren’t enough details in a method section to provide a comprehensive “recipe” that you can follow, which is somewhat frustrating when you’re trying to learn how others use a given stats method. It would seem that my PhD examiners may be in for some pretty tedious, if thorough, methodological sections! But at least I feel like I’m putting in the effort to use the statistical technique the data call for and to understand what I’m doing.