Richard Miniter Versus the Lancet
Pajamas Washington DC editor Richard Miniter interviews Lancet study author Professor Gilbert Burnham at Pajamas Media. I am somewhat disappointed in Burnham, who seemed to have no other answer to Miniter's specific questions other than to assert, time and again, 'but this is what our research methodology told us'. A sample of the interview is given below:
PajamasMedia: The Lancet study uses a baseline mortality rate (the rate during Saddam years) of 5.5/1000 – almost half of the mortality rate of Europe. The mortality rate in the EU is 10.10/1000. Given Europe’s excellent health care, public health infrastructure and, lack of war in the past 60 years, how is it possible that Iraq’s baseline is half that of the EU? Are you simply relying on pre-war publications or was the baseline itself generated by interviews with random clusters?
Burnham: This was a ‘cohort’ study, which means we compared household deaths after the invasion with deaths before the invasion in the same households. The death rates for these comparison households was 5.5/1000/yr.
What we did find for the households as a pre-invasion death rate was essential the same number as we found in 2004, the same number as the CIA gives and the estimate for Iraq by the US Census Bureau.
Death rates are a function of many things—not just health of the population. One of the most important factors in the death rate is the number of elderly in the population. Iraq has few, and a death rate of 5.5/1000/yr in our calculation (5.3 for the CIA), the USA is 8 and Sweden is 11. This is an indication of how important the population structures are in determining death rates. (You might Google ‘population pyramid’ and look at the census bureau site—fascinating stuff.)
PajamasMedia: During the same period, Iraq is at war with Iran and itself. Public-health infrastructure was poor, although perhaps not as poor as today. Does it seem plausible to you that the baseline (or pre-war) mortality rate is accurate?
Burnham: Yes as above. Yes as being the right number, and Yes as what we need it for—comparisons in the same households before the war.
A moment's thought would have convinced Burnham that, except from a very narrow definitional point of view, a person contributes to the death rate whether he happens to die young or old. The death rate per 100,000 over the course of a person's life will be reflected in life expectancy. The higher the death rate per 100,000 the lower the average life expectancy. The lower the death rate the higher the life expectancy. Europe's population is old because it has a low death rate per 100,000. In fact, if the death rate were zero they would live forever.
But, one might say that however nonsensical his definition, Burnham is using comparable pre-OIF statistics against the post-OIF statistics and whatever problems the underlying concept of his definition may have the comparison is apples to apples. But how does he measure his apples?
PajamasMedia: You conducted interviews in 47 clusters, 12 of them were in Baghdad, 2 were in Basra, and 3 were in Anbar. Approximately 25% of the estimate comes from Baghdad (only 21% of the population in Baghdad). This seems disproportionate. Is it possible that you over-sampled “hot zones” relative to population?
Burnham: We used the 2004 UNDP/MoP estimates for governorates. We divided the total population by the number of clusters, and then moved on a systematic way through the population assigning clusters proportionate to the population numbers we were using. ... There are always chances that sampling was done in more hot spots, but there is an equal chance that, with a natural human tendancy to self-preservation might cause sampling to be the other way to unconsciously sample in cool spots where one might be safer.
Burnham is apparently using a kind of stratified sampling methodology. He uses the UNDP/MoP to identify different population groups within Iraq and then goes out and measures groups that he thinks correspond to these populations, obtains his statistic and then extrapolates it according to the known weights. This means that if he wrongly identifies his sample with the wrong population groups both his weights and results will be wrong as well. But as you can see he doesn't know whether he is sampling hot spots -- the unrepresentative spots -- at all. In fact, he argues that they may have been "cool spots". But he has to know: because unless he knows, given the methodology he employs, his survey will be useless. Let's take an example.
Trees grow better in a valley where they have abundant water and soil and grow more poorly on hilltops where the soil is thin and dry. Your job is to estimate the timber in the forest. You have an estimate of the proportion of hilltops to valleys. A man is put blindfolded into a forest to inventory it and when he is already in the midst of it the blindfold is removed to measure the trees but he can't see whether he is on a hill or a valley. Can he extrapolate the trees volumes without knowing whether he was on a hill or a valley? What value would a timber inventory have absent that knowledge? If Burnham didn't know whether he was in a cool or hot spot, he would lack the most critical piece of information to make his study work.
As I said, I'm disappointed.