The Amateur Technocrat VII

Experiment, Observation, Econometrics

The blogosphere’s a fun place isn’t it? Always throwing up a post somewhere where the argument is so irritatingly and egregiously wrong that it niggles and niggles at you until the only way to get rid of the mounting irritation is to sit down and write as thorough a refutation as you can. And on a Sunday too!

Let’s start with a not so imaginary experiment. Suppose you want to test the effect of two newly discovered drugs – Viaglis and Cialgra – on the sexual behaviour of male rats. After a little thought (and it wouldn’t take a biological scientist much thought to get to this stage) we decide that the way to do it is to take a few adult male rats, give some of them a shot of Viaglis, another group a shot of Cialgra, a third lot (our control group) nothing at all, put each of them in with a female rat and see how often they try it on. But that’s just the easy part.

Once we’ve decided on this approach, we now have to put in a lot of thought about how to keep our ceteris genuinely paribus – we have to identify all the other factors that influence our test variable – observed root attempts – so that we can control for those factors. Here’s a far from comprehensive list:

  • The sexual maturity of the experimental male rats;
  • The sexual maturity of the female rats;
  • Oestrus in the female rats;
  • Experimenter ignorance of rat sexual behaviour.

So we now have some serious decisions to make – for example, do we want to use sexually mature rats, or sexually immature ones? Well, what are we testing for – heightened sexual activity in already sexually active rats (or rats who would be sexually active if the animal house gave them the opportunity) or the capacity of Viaglis and Cialgra to induce precocious sexual behaviour? There are other complications – other issues of experimental design – that arise from the the first three factors. Identifying most of those I leave as an exercise for readers and commenters.

A few remarks are warranted on the last, before we move on. To overcome experimenter ignorance, we can find experimenters who know a lot about rat behaviour – but then the knowledge and experience of our experimenters enters the experiment as a very big, very uncontrolled, variable in our experimental results. Much better is to identify a simple, easily recognisable sexual behaviour – call it a mount – that is easily recognised by any observer after they have seen it once. In addition, we give our observers a stop watch, and tell them that they are to record not only the occurrence of a mount, but its duration as well. Hidden away in our own notes of the experimental design (in some secure place where the observers can’t read about it until after we’ve published) is the stipulation that only mounts that last two seconds or longer will be counted as experimental data.

At the end of the process – assuming that our ceteris stays paribus throughout – that all the rats we get from the animal house are within spec, for example – we’ll have a collection of data we can use to produce some impressive tables and graphs. Or we can use it to produce informative tables and graphs, such as the hypothetical one that I knocked up for illustration (below). You might not find it particularly informative (sorry about that) but someone who went through a Pharmacology major at roughly the time I did – last century – probably would*.

Hypothetical DR plot

Now let’s tackle the same question – how do Viaglis and Cialgra affect rat sexual behaviour – with an econometric approach, similar to the econometric approach deployed in a recent paper (PDF) on the question of whether using a mobile phone while you drive increases your risk of a road traffic “accident”. The paper is published on the web-site of the Brookings Institute (link via Nick Gruen at Club Troppo).

Since we’re econometricians – not biologists – we obviously don’t have the skills and the knowledge to design an experiment, so we’ll look for one that’s been done for us.

Somehow, we learn of a “natural experiment” where an on-line black-market for Viaglis and Cialgra emerged, and rat-fanciers were using to (so they thought) increase the sexual activity of their male stud rats, to produce more rat kits for the pet-shop trade, rat showing etc.

We might not know how, or whether, Viaglis and Cialgra, affect rat behaviour, but we are capable of reasoning thus:

  • If Viaglis and Cialgra increase sexual activity in male rats, we would expect to see more rat matings. Rat matings result in rat procreation therefore
  • We would expect to see an increase in the reproduction rate of pet rats kept by the rat-breeders in this market. This increase in the reproduction rate would result in an increase in the supply of pet rats for sale. We can find out whether this happened by checking the levels of retail stock in pet rats while this black market in Viaglis and Cialgra was operating. Furthermore
  • An increase in the supply of pet rats, given the laws of supply and demand we all learnt in Economics 101 should lead to a decrease in the retail price of pet rats. We can check for this effect too. Furthermore
  • A decrease in the retail price of pet rats, given all that stuff we learnt about substitution, should lead some consumers to buy pet rats, rather than dogs, cats, budgies or hamsters, so there’s another source of data we can look at.

And so on. With a bit of creative thinking we can find plenty of statistical and econometric tests we can apply that will identify a very remote, very downstream effect of feeding male rats Viaglis and Cialgra. A bit more creative thinking – and it wouldn’t take any creative thinking that wasn’t beyond our capacities as econometricians – and we could dress our study up in enough pseudo-scientific finery that it might get past a research grants committee and qualify as a piece of “objective, government-funded research”. That would be so much better than having to do the work under the auspices of a private research foundation, noted for its political advocacy, and thus raise the suspicion that our study was no more than “advocacy research”.

So out we go, and perform our elaborate study, and find that according to all of the indicators we have identified, Viaglis and Cialgra have no effect whatsoever on the sexual behaviour of male rats – no increase in the retail supply of pet-rats at discounted prices, no significant effect on sales of substitute products such as puppies, kittens, budgies and hamsters, and so on. Nothing. What does surprise us is that when we offer the paper describing our results to the refereed journals, none of them want to touch it. And that other scientists and scholars persist in believing that Viaglis and Cialgra do affect the sexual behaviour of male rats and we find ourselves in a lot of pointless – and humiliating – discussions along the following lines:

Us: but if Viaglis and Cialgra have this effect on male rats, why didn’t we see any changes in all the economic indicators we studied?

Biologist (exasperated): Because you’re a bunch of idiots with no idea of how to do real research and your study was all ceteris and no paribus.

Us: but we controlled for all that in our statistical analysis, our multi-variate regressions, our …

Biologist (still exasperated): You didn’t control for anything, moron. Now piss off – I want a chance to talk to that astrologist over there – the one the gorgeous bum.

There are, perhaps, two consolations we can offer ourselves in this situation. First that even though we haven’t managed to convince any of the scientists whom we’d like to regard as our peers that they’re wrong about Viaglis and Cialgra and we’re right, there’s probably an advocacy web-site out there where our work will be welcomed as the ground-breaking, impartial government-funded study it really is. Second, that as econometricians, it’s much more likely that we’ll finish the evening in bed with that astrologist over there (the one with the gorgeous bum) than any biologist.

* This is pure laziness on my part. Still, for those who do want an explanation of the graph, it’s a hypothetical pair of dosage/response plots showing how Viaglis (V) and Cialgra affect mounting in male rats. Viaglis (V) to the left, is more efficacious at all doses than Cialgra (C) to the right. The horizontal, x-axis, uses a logarithmic scale because that’s just the way it’s done, or was done back in the day. There are – or were – good reasons for that which we needn’t go into here, OK?


« profile & posts archive

This author has written 29 posts for Larvatus Prodeo.

Return to: Homepage | Blog Index

17 responses to “The Amateur Technocrat VII”

  1. Ambigulous

    thank you Gummo,
    youse have outdid yourself here. And no mention of rattus nonveritas?

    I confidently predict that both the sexual biology of the rat AND econometrics, will never be the same ever again. The world has changed. These brazen attacks on the rat rooting labs (as the technical phrase goes) and even worse, these shocking assaults on the citadels of econometrics, show just what vicious and determined enemies we face: they hate us for our ECONOMETRICS.
    The cads. The zealots. Mired in medieval obscurantism.

  2. philiptravers

    A picture is worth a thousand words..Thus Spake Jack Little of World Championship Wrestling.And seeing timelapse video and even internet camera now can observe remotely.The PDF download would have to be a well constructed propaganda piece dressed up in all the refinements necessary to do the work, and, thus give less ammo to the real observations involved in the basic of econmetrics which were!? The Time and Motion man. Men ,as plural. Try Deprenyl instead of Viagra!?

  3. patrickg

    Hmm. This would have been more interesting if you had used the actual case/s in question.

  4. Gummo Trotsky

    Hmm. This would have been more interesting if you had used the actual case/s in question.

    No it wouldn’t. It wouldn’t have been written at all.

  5. su

    That was very funny. How on earth they managed to get that cell phone study off the ground I cannot imagine. What a crock. A study investigating cell phones and crashes which neatly avoids the necessity of investigating cell phone use in crashes. Next; a study that finally refutes the suggestion of an association between cell phones and tumorgenesis by tracking the price of radioactive isotopes.

  6. Dave Bath

    * We would expect to see an increase in the reproduction rate of pet rats kept by the rat-breeders in this market.
    Only if the “receptors” were already not “saturated”.

    I looked at the study you mention, and the line of reasoning to price is poor. (Peak v non-peak is a poor choice as the most “critical” calls to answer come during peak hours for both phones and cars – “when will you get here?”).

    Personally, if hand-eye co-ordination and reaction times are a decent measure of driving incapacity (and they are used to figure out alcohol blood levels by getting a group of pharmacology students drunk in driving simulators), then a quicker test would be to measure decrease in scores of computer gamers when they are talking on mobile phones.

  7. zoot

    Personally, if hand-eye co-ordination and reaction times are a decent measure of driving incapacity (and they are used to figure out alcohol blood levels by getting a group of pharmacology students drunk in driving simulators), then a quicker test would be to measure decrease in scores of computer gamers when they are talking on mobile phones.

    I believe it’s already been done, but it may just be my imagination. Something on the Science Show if I remember correctly.

  8. Andrew Leigh

    I’m with patrickg. The Brookings paper is sophisticated enough that arguing by metaphor doesn’t get you far. Their point seems pretty simple: thanks to non-linear pricing schedules, we know call volume rises at 9pm. If there’s a big effect of cellphone calls on accidents, then accidents should jump at that time as well. Since the accident rate doesn’t seem to jump at 9pm, the effect of cellphone use on accidents is probably pretty small. Of course, it’s conceivable that the price elasticity of drivers is different, but this seems unlikely to me.

    The reaction time approach only works if no driver ever adjusts his/her behaviour while on the phone. If some drivers behave differently while on the phone, then this approach will only give you an upper bound.

    Also, what makes you think that the authors got industry money? It looks to me as though it’s funded through Berkeley’s “Institute of Business and Economic Research”, which hardly seems a stooge for the cellphone industry.

  9. Gummo Trotsky

    Here’s one very obvious exogenous factor that I’ll have to check through the original paper, to see if it’s been accounted for in the paper in question: avoidance behaviour by other drivers and other road users.

    I don’t drive myself, but when I’m about to cross at a green light and some hoon comes tear-arsing through the intersection because he hit the accelerator when he saw the light change to amber, I’m usually alert enough to step back when my foot hits the road surface. When I’m a passenger, there are plenty of opportunities to observe the driver adjusting his/her driving to deal with the fact that we’re stuck behind an idiot who’s not paying attention to the road and other drivers. And so on.

    As for the sophistication of the paper’s analysis – well sophisticated and sophistical share the same Greek root you know.

  10. Shaun

    thanks to non-linear pricing schedules, we know call volume rises at 9pm. If there’s a big effect of cellphone calls on accidents, then accidents should jump at that time as well.

    I don’t follow the logic here. The volume of traffic at this time of night is far less than during the day and or/peak hours. Also, many who may take advantage of such calls would not necessarily be on the road. It also doesn’t reflect people who SMS (which would not be subject to the constraints of peak/off peak pricing) when driving.

    I’d say more people would receive and make mobile calls around business hours (give or take a few each side).

  11. Gummo Trotsky

    Here’s the basic reasoning of that Brookings paper, shorn of all the distracting wool:

    If my absurd extrapolation from your experimental findings is correct I’d expect to see something like this .

    But I don’t see anything like this at all. So …

    Your findings must be wrong.

  12. Andrew Leigh

    GT, it’s possible that drivers suddenly become extra-careful after 9pm because they’re aware that the non-linear change in cellphone pricing schedules will increase cellphone use. But this seems pretty unlikely to me.

    Shaun, they’re not comparing day and night; they’re comparing 8-9pm and 9-10pm. Plus they do a weekday/weeknight comparison, so we can account for the fact that accident load steadily declines through the evening. You’re of course right that the danger could all be from SMSing drivers, but that’s not what most people contend is the primary risk of cellphone use.

    The extrapolation isn’t absurd. They show from other data that cellphone use jumps 20-30% at 9pm.

  13. su

    There are so many possible confounds in this study that it is just junk. For a start they assume that an increase in call volume necessarily translates to an equivalent jump in call volume from cars. That alone is a major confound. Perhaps every single extra call is made from home or restaurants etc; it is after 9pm after all. The point it is that it is an unknown and therefore uncontrolled for variable.

  14. Gummo Trotsky

    Andrew,

    My remark on avoidance behaviours didn’t posit any awareness by drivers “that the non-linear change in cellphone pricing schedules will increase cellphone use.”

    More the sort of casual observation you’ll sometimes hear described in conversations between drivers and passengers “Jeez, there’s a lot of bloody dickheads out on the road tonight, where did they all come from?”

    “Dunno mate – maybe it’s thanks to the non-linear change in the mobile phone pricing schedule.”

  15. Shaun

    Andrew,

    The other factor not taken into account is the demographics of those who use mobiles. I’d like to see if there is skew towards younger users after 9:00pm who are home not about and about. Also, with less cars on the road after 9:00pm, I’d say that affects the accident rate as well.

    I’m with su that to many variables are not taken into account and the study is flawed.

  16. Bill O'Slatter

    No I don’t think the study is flawed : the confidence levels are pretty tight. I think iit makes good use of a natural randomised trial. That one of the greats of statistics Tibshirani found an effect however would need some extra explanation. ( like Fisher finding no effect of smoking)
    The effect of mobile phone use ins certain subgroups ( e.g. teenagers) may not be so indifferent.

  17. patrickg

    See now this debate is so much more interesting than argle-bargle about rats. More interesting, and funnier too!