Experiment, Observation, Econometrics
The blogosphere’s a fun place isn’t it? Always throwing up a post somewhere where the argument is so irritatingly and egregiously wrong that it niggles and niggles at you until the only way to get rid of the mounting irritation is to sit down and write as thorough a refutation as you can. And on a Sunday too!
Let’s start with a not so imaginary experiment. Suppose you want to test the effect of two newly discovered drugs – Viaglis and Cialgra – on the sexual behaviour of male rats. After a little thought (and it wouldn’t take a biological scientist much thought to get to this stage) we decide that the way to do it is to take a few adult male rats, give some of them a shot of Viaglis, another group a shot of Cialgra, a third lot (our control group) nothing at all, put each of them in with a female rat and see how often they try it on. But that’s just the easy part.
Once we’ve decided on this approach, we now have to put in a lot of thought about how to keep our ceteris genuinely paribus – we have to identify all the other factors that influence our test variable – observed root attempts – so that we can control for those factors. Here’s a far from comprehensive list:
- The sexual maturity of the experimental male rats;
- The sexual maturity of the female rats;
- Oestrus in the female rats;
- Experimenter ignorance of rat sexual behaviour.
So we now have some serious decisions to make – for example, do we want to use sexually mature rats, or sexually immature ones? Well, what are we testing for – heightened sexual activity in already sexually active rats (or rats who would be sexually active if the animal house gave them the opportunity) or the capacity of Viaglis and Cialgra to induce precocious sexual behaviour? There are other complications – other issues of experimental design – that arise from the the first three factors. Identifying most of those I leave as an exercise for readers and commenters.
A few remarks are warranted on the last, before we move on. To overcome experimenter ignorance, we can find experimenters who know a lot about rat behaviour – but then the knowledge and experience of our experimenters enters the experiment as a very big, very uncontrolled, variable in our experimental results. Much better is to identify a simple, easily recognisable sexual behaviour – call it a mount – that is easily recognised by any observer after they have seen it once. In addition, we give our observers a stop watch, and tell them that they are to record not only the occurrence of a mount, but its duration as well. Hidden away in our own notes of the experimental design (in some secure place where the observers can’t read about it until after we’ve published) is the stipulation that only mounts that last two seconds or longer will be counted as experimental data.
At the end of the process – assuming that our ceteris stays paribus throughout – that all the rats we get from the animal house are within spec, for example – we’ll have a collection of data we can use to produce some impressive tables and graphs. Or we can use it to produce informative tables and graphs, such as the hypothetical one that I knocked up for illustration (below). You might not find it particularly informative (sorry about that) but someone who went through a Pharmacology major at roughly the time I did – last century – probably would*.
Now let’s tackle the same question – how do Viaglis and Cialgra affect rat sexual behaviour – with an econometric approach, similar to the econometric approach deployed in a recent paper (PDF) on the question of whether using a mobile phone while you drive increases your risk of a road traffic “accident”. The paper is published on the web-site of the Brookings Institute (link via Nick Gruen at Club Troppo).
Since we’re econometricians – not biologists – we obviously don’t have the skills and the knowledge to design an experiment, so we’ll look for one that’s been done for us.
Somehow, we learn of a “natural experiment” where an on-line black-market for Viaglis and Cialgra emerged, and rat-fanciers were using to (so they thought) increase the sexual activity of their male stud rats, to produce more rat kits for the pet-shop trade, rat showing etc.
We might not know how, or whether, Viaglis and Cialgra, affect rat behaviour, but we are capable of reasoning thus:
- If Viaglis and Cialgra increase sexual activity in male rats, we would expect to see more rat matings. Rat matings result in rat procreation therefore
- We would expect to see an increase in the reproduction rate of pet rats kept by the rat-breeders in this market. This increase in the reproduction rate would result in an increase in the supply of pet rats for sale. We can find out whether this happened by checking the levels of retail stock in pet rats while this black market in Viaglis and Cialgra was operating. Furthermore
- An increase in the supply of pet rats, given the laws of supply and demand we all learnt in Economics 101 should lead to a decrease in the retail price of pet rats. We can check for this effect too. Furthermore
- A decrease in the retail price of pet rats, given all that stuff we learnt about substitution, should lead some consumers to buy pet rats, rather than dogs, cats, budgies or hamsters, so there’s another source of data we can look at.
And so on. With a bit of creative thinking we can find plenty of statistical and econometric tests we can apply that will identify a very remote, very downstream effect of feeding male rats Viaglis and Cialgra. A bit more creative thinking – and it wouldn’t take any creative thinking that wasn’t beyond our capacities as econometricians – and we could dress our study up in enough pseudo-scientific finery that it might get past a research grants committee and qualify as a piece of “objective, government-funded research”. That would be so much better than having to do the work under the auspices of a private research foundation, noted for its political advocacy, and thus raise the suspicion that our study was no more than “advocacy research”.
So out we go, and perform our elaborate study, and find that according to all of the indicators we have identified, Viaglis and Cialgra have no effect whatsoever on the sexual behaviour of male rats – no increase in the retail supply of pet-rats at discounted prices, no significant effect on sales of substitute products such as puppies, kittens, budgies and hamsters, and so on. Nothing. What does surprise us is that when we offer the paper describing our results to the refereed journals, none of them want to touch it. And that other scientists and scholars persist in believing that Viaglis and Cialgra do affect the sexual behaviour of male rats and we find ourselves in a lot of pointless – and humiliating – discussions along the following lines:
Us: but if Viaglis and Cialgra have this effect on male rats, why didn’t we see any changes in all the economic indicators we studied?
Biologist (exasperated): Because you’re a bunch of idiots with no idea of how to do real research and your study was all ceteris and no paribus.
Us: but we controlled for all that in our statistical analysis, our multi-variate regressions, our …
Biologist (still exasperated): You didn’t control for anything, moron. Now piss off – I want a chance to talk to that astrologist over there – the one the gorgeous bum.
There are, perhaps, two consolations we can offer ourselves in this situation. First that even though we haven’t managed to convince any of the scientists whom we’d like to regard as our peers that they’re wrong about Viaglis and Cialgra and we’re right, there’s probably an advocacy web-site out there where our work will be welcomed as the ground-breaking, impartial government-funded study it really is. Second, that as econometricians, it’s much more likely that we’ll finish the evening in bed with that astrologist over there (the one with the gorgeous bum) than any biologist.
* This is pure laziness on my part. Still, for those who do want an explanation of the graph, it’s a hypothetical pair of dosage/response plots showing how Viaglis (V) and Cialgra affect mounting in male rats. Viaglis (V) to the left, is more efficacious at all doses than Cialgra (C) to the right. The horizontal, x-axis, uses a logarithmic scale because that’s just the way it’s done, or was done back in the day. There are – or were – good reasons for that which we needn’t go into here, OK?