In October 2013 a new study was released in the PLOS
Medicine journal estimating the number of deaths following the 2003 invasion of
Iraq. It attempted to avoid and make up for some of the criticism of previous
surveys that looked at fatalities in the country. These included the 2004 and
2006 Lancet papers, the 2004 Iraq Living Conditions Survey, the 2007 Opinion
Business Research (ORB) poll, and the 2008 Iraq Family Health Survey. The new
report estimated 460,000 excess deaths occurred after the fall of Saddam
Hussein. It didn’t give a figure for all violent deaths, but did say that for
adults aged 15-60 132,000 died because of violence from 2003-2011, which is
very similar to Iraq Body Count and figures recorded by the U.S. military. The
Iraqi Family Health Survey and the Iraq Living Conditions were both within the
range of the new one as well, while the two Lancet reports and the ORB
questionnaire were far outside of it. While no survey can be authoritative the
new PLOS one at least confirms that a few of the earlier estimates were
capturing some of the death and destruction released by the Iraq War, while
largely repudiating the Lancet articles.
The PLOS paper set out to avoid some of the criticism laid
at previous Iraq death surveys on Iraq, and wanted a new up to date estimate
since the last one was done in 2007. The new report questioned 2,000 randomly
selected households in 100 clusters across the country. It used a random
selection process to generate which clusters were to be surveyed. It then
picked one house as the starting point, and then interviewed that one plus 19
adjacent ones. It wanted to use double the number of clusters as the 2004 and
2006 Lancet studies and a more sophisticated random selection process to reduce
the chance of missing high and low conflict related death areas. This was due
to one critique leveled at the two Lancet articles called the “main street bias.” That argued that pollsters went to main streets and then polled the
cross streets, which had the highest chance of violence
occurring, and thus led to high death estimates. The Lancet papers were questioned
about how exactly they went about deciding what areas were to be surveyed as
well.
To prepare for the survey the team met in Kurdistan in March
2011 to go over procedures, then trained the survey teams, which were made up
of Iraqi medical doctors with experience in polling. The groups set out across
the country from May to July, and then everyone reconvened in September to go
over the preliminary findings. Households were questioned about all the births
and deaths they experienced since 2001. When a passing was reported a death
certificate was requested. Those were produced 74% of the time, and a
certificate was reported but not seen 17% of the time. For war related
fatalities specific causes and who was responsible were asked for. Adults were
then queried about their siblings’ experiences. That resulted in data for
another 24,759 people. A supervisor then went to one randomly selected
household in each cluster to repeat the survey as a quality check. In the end
there was a 98.55% response rate among the 2,000 households. Training and
supervision are important facets to conducting a scientific survey. The fact
that violence had decreased considerably by 2011 when the poll was conducted
also meant that there was access to almost every part of the country. Only five
out of the 2,000 households for example could not be questioned because of
threatening or hostile behavior. Raising questions about siblings also greatly increased
the available data that could be analyzed.
Another part of the study was an attempt to account for
Iraq’s displaced and refugee population. The group consulted secondary sources
to estimate how many refugees there were. A number of calculations were then
made with that figure to come up with a number of deaths amongst refugees. The
authors knew that they missed many possible excess deaths amongst this group,
which was one shortcoming they acknowledged in their findings. At the same time
a large number of Iraqis have returned to their homes or have settled in new
parts of the country since violence subsided in 2008, so the poll might have
captured more of this group then they believed.
To come up with their findings they broke up their data by time
periods. They estimated a crude death rate by counting the number of deaths in
all households in each time period, and then divided that by the number of
persons and years they lived within those eras. The baseline crude death rate
was set for the period before the war from 2001 to March 2003. To calculate the
war time deaths they used the differences between the crude death rate for each
time period and the crude death rate for the pre-war era. For the siblings data
they knew there were biases and tried to adjust for that. Another issue was
that households might not have been able to remember everything accurately
since they had to recall events from several years ago.
The final results found almost half a million deaths since
2003. The pre-war crude death rate from 2001 to 2003 was 2.89 per 1,000. The
wartime rate was 4.55 per 1,000 from March 1 2003 to June 30, 2011. By
multiplying those rates by the estimated population they came up with
approximately 405,000 excess deaths with a range of 48,000 to 751,000 after
2003. Another 55,000 fatalities were estimated for the refugees for a total of
460,000 deaths in the post-invasion period. A total number for violent deaths
was not given, but a rate for adults aged 15-60 was estimated at 132,000 during
the war. Overall, the author’s believed that 60% of the excess deaths were a
direct result of the war. The other 40% came from the collapse of
infrastructure and services that occurred after the invasion. Like previous
surveys a major problem with calculating crude death rates was that Iraq lacks
up to date census data. That means those figures could go up and down depending
upon how many people tne report believed the country had.
The survey found a range of causes for the deaths. Amongst
violent ones gunshots not bombs was the most common at 63%, followed by car
bombs, 12%, other explosions, 9%, other war injuries, 9%, and airstrikes, 7%.
Those most responsible were Coalition Forces, 35%, militias and insurgents,
32%, others/unknown, 21%, criminals, 11%, and Iraqi forces, 1%. For nonviolent
deaths cardiovascular conditions were at 47%, then infant or childhood
deaths/injuries, 12%, and chronic illness and non-war injuries were both at
11%. The prevalence of shootings showed how bloody the civil war period was.
The deadliest period for Iraqis was from 2005-2007 when Baghdad and its
environs were the center of the fighting. There, militias and insurgents were
fighting for control of communities block by block. Bombings might have grabbed
the headlines, but the sectarian war was at the street level involving assault
rifles, pistols, and machine guns. That was also shown in the fact that the
militias and insurgents were responsible for almost as many deaths as the
Coalition, which fought pitched battles such as the two for Fallujah, the three
against the Mahdi Army, as well as various other major campaigns. The
non-violent deaths were blamed on the war as well by the authors. They wrote
that the collapse of infrastructure and essential services like electricity and
clean water were a result of the invasion and its aftermath.
 |
| (Daily Beast) |
 |
| (Daily Beast) |
There have been five previous surveys in Iraq trying to
estimate the number of deaths since 2003. Only two dealt with violent
casualties, while the other three looked at all possible fatalities. The
reports found wide differences in mortality rates, and all came in for some
sort of criticism such as potential bias in sampling, wide ranges of
uncertainty intervals, small sample sizes, disputes over statistical methods,
choice of population figures, incomplete questioning, and plausibility of
results. The new PLOS Medicine paper did a comparison of crude death rates per
1,000 with some of the previous reports. The 2004 and 2006 Lancet polls were in
the upper range of the PLOS pre-war rates, and far above it for the wartime
period. The 2004 Lancet had a pre-war mortality rate of 5.0 per 1,000 from
2002-2003 and 12.3 per 1,000 from 2003-2004, while the 2006 one recorded
5.5 per 1,000 from 2002-2003 going up to 13.3 from 2003-2006. That compared
to the PLOS’s 2.89 per 1,000 from 2001-2003 and 4.5 per 1,000 from 2003-2011.
Another outlier was the 2007 Opinion Research Business poll that estimated a
violent death rate of 10.3 per 1,000 in 15 of 18 provinces from 2003-2006. The
Iraq Living Conditions Survey and the Iraq Family Health Survey on the other
hand were both within the range of the new survey’s pre and post invasion
rates. The Iraq Family Health Survey for instance reported 5.31 deaths per
1,000 from 2003-2006. Those similarities and differences were even starker
when comparing estimated death counts. The 2004 Iraq Living Conditions survey
estimated 24,000 deaths from 2003-2004. The first Lancet covered the same
time period, and was four times higher at 98,000 killed from 2003-2004. That
did not include Anbar, so the authors believed the total figure was much larger.
The Iraq Family Health Survey released in 2008 calculated 151,000 deaths from
2003-2006. The second Lancet paper was for the same years and was more than
four times that at 654,965. Even farther apart from all the rest was the
Opinion Reach Business survey that estimated 1,033,000 deaths from 2003-2007. The PLOS report calculated figures for eight years, and found 460,000
fatalities. Its authors came up with three hypotheses for why their numbers
were different than the others. Those were sampling differences, recall bias,
and non-sampling errors. PLOS used different sampling methods from the two
Lancet and Iraqi Family Health surveys. The Iraq Family Health Survey also
skipped 115 of its clusters for security reasons and consulted Iraq Body Count
to calculate how any deaths might have occurred in those areas. Finally, households were asked to recall events about
themselves going back ten years, but also for their siblings in the newest
poll, which probably led to underreporting of deaths as some were missed. The
PLOS team also believed they missed many deaths from the refugee and displaced
populations. Although the new report compared itself to the five previous ones,
it seemed like it was going out of its way to avoid many of the criticisms laid
at the Lancet articles. Although still widely accepted, the Lancet estimates
for deaths have faced intense criticism by academics. Many thought the 2004
and 2006 papers came up with far too high estimates. The new survey is just
one more piece of evidence that those two polls were off the mark. In
comparison, the Iraq Living Conditions and Iraq Family Health Survey appear to
be more on point.
Comparison Of
Pre-Iraq War Crude Death Rates
PLOS Medicine 2.89 per 1,000 2001-2003
2004 Lancet 5.0 per 1,000 2002-2003
2006 Lancet 5.5 per 1,000 2002-2003
Comparison Of
Post-Invasion Crude Death Rates
PLOS Medicine 4.5 per 1,000 2003-2011
Iraq Family Health Survey 5.31 per 1,000 2003-2006
Opinion Research Business 10.3 per 1,000 2003-2006
2004 Lancet 12.3 per 1,000 2003-2004
2006 Lancet 13.3 per 1,000 2003-2006
Comparison Of
Estimated Deaths
Iraq Living Conditions 24,000 2003-2004
2004 Lancet 98,000 2003-2004
Iraq Family Health Survey 151,000 2003-2006
2006 Lancet 654,965 2003-2006
Opinion Research Business 1,033,000 2003-2007
PLOS Medicine 460,000 2003-2011
 |
| Chart shows that the Iraq Family Health Survey (IFHS) was within the middle of the range of the crude death rate in the new PLOS survey while the 2004 Lancet (Roberts 2004) and 2006 Lancet (Burnham 2006) were at the top in their pre-war crude death rate prediction, and far outside of the range of the wartime estimated rate (PLOS Medicine) |
The new survey has received a large amount of press as well
as received critiques and gained supporters. John Tirman who commissioned the
Lancet study wrote that people that criticized that report were doing a
disservice by trying to play down the number of people killed by the Iraq War. He pointed to the PLOS paper as more evidence of the high cost of the U.S.
invasion. Tom Paulson in Humanosphere claimed that the new survey verified the
2006 Lancet paper. Those two ignored the fact that both the Lancet crude
death rate estimates were outside the range of the new survey, and that 460,000
killed over eight years is not close to the second Lancet’s 654,000 over just
three. Josh Dougherty an analyst for Iraq Body Count and University of London
Economics Professor Michael Spagat both pointed that out. Dougherty noted that
the PLOS survey only found 76 violent deaths from 2003-2006 compared to 300 by
the 2006 Lancet. He also estimated that if the new report was limited to
just the 2nd Lancet period it would come up with less than 200,000
deaths, more than three times lower. Spagat critiqued the wide confidence
interval in the new survey of 48,000-751,000 stating that probably made their
estimated 460,000 killed too high rather than too low as the authors have repeatedly said to the media. Joshua Keating in Slate and John Rentoul in the Independent each argued that the PLOS report is roughly
congruent to Iraq Body Count and the U.S. military’s figures released by
Wikileaks of 100,000-126,000 killed. Finally Dr. Prabhat Jha of the Center for
Global Health Research in Toronto was quoted in the Los Angeles Times pointing
out that the lack of an accurate census for Iraq could mean that the estimated
number of deaths could be lower if the population was higher than the 32
million used in the survey’s calculations. With one of the fastest growing
populaces in the Middle East and North Africa there’s a good chance of that.
Overall, those that have used the PLOS paper to support Lancet have not read
the new report. It is more of a refutation of the 2004 and 2006 Lancets than a
confirmation. There’s a chance that the estimated deaths from the new report
are actually lower, which would make it even farther away from the two Lancets.
The PLOS Medicine article adds another important piece to
the argument over how many Iraqis died after the 2003 invasion. First, it is
another paper that shows that the two Lancet reports were far too high in their
estimates for deaths following the fall of Saddam Hussein. That was previously
shown with the Iraq Living Conditions, the Iraq Family Health Survey, and
various academic criticisms. Second, it shows that Iraq Body Count’s
method of recording deaths using media reports while not perfect does a very
good job as its estimate for violent deaths is very close to the PLOS one for
adults. Finally, it highlights just how deadly the sectarian war was in Iraq.
Today bombings by Al Qaeda in Iraq are causing the majority of fatalities in
the country’s deteriorating security situation. If shootings were to suddenly
increase that would be a good sign that a civil war is breaking out again as
insurgents, militias, and the security forces battle for control. In a war,
especially like the one in Iraq where the government broke down for a few years
a definitive number for fatalities will never be determined. It’s only through
various surveys and studies that a rough estimate can be made. The PLOS report
adds to this very important debate on the human cost of the Iraq invasion.
SOURCES
Brownstein, Joseph, “Iraq war
claimed half a million lives, study finds,” Al Jazeera, 10/15/13
Burnham, Gilbert, Doocy, Shannon, Dzeng, Elizabeth, Lafta, Riyadh,
Roberts, Les, “The Human Cost of the War in Iraq, A Mortality Study,
2002-2006,” Bloomberg School of Public Health Johns Hopkins University, School
of Medicine Al Mustansiriya University, 9/26/06
Cole, Juan, “The American Genocide
Against Iraq: 4% of Population Dead as result of US sanctions, wars,” Informed
Comment, 10/17/13
Dougherty, Joshua, Iraq Body Count, private conversations,
October 2013
Hagopian, Amy, Flaxman, Abraham,
Takaro, Tim, Al Shatari, Sahar Esa, Rajaratnam, Julie, Becker, Stan, Levin-Rector,
Alison, Al-Yasseri, Berq Hadi, Weiss, William, Murray, Christopher, Burnham,
Gilbert, “Mortality in Iraq Associated with the 2003-2011 War and Occupation:
Findings from a National Cluster Sample Survey by the University Collaborative
Iraq Mortality Study,” PLOS Medicine, October 2013
Iraq Family Health Survey Study Group, “Violence-Related
Mortality in Iraq from 2002 to 2006,” New England Journal of Medicine, 1/31/08
Johnson, Neil, Spagat, Michael, Gourley, Sean, Onnela,
Jukka-Pekka, and Reinert, Gesine, “Bias in Epidemiological Studies of Conflict
Mortality,” Journal of Peace Research, September 2008
Keating, Joshua, “Half a Million
Deaths is a Statistic,” Slate, 10/18/13
Morin, Monte, “Study estimates
nearly 500,000 Iraqis died in war,” Los Angeles times, 10/15/13
Paulson, Tom, “The great Iraq
death toll debate: Putting war on the global health agenda,” Humanosphere,
10/18/13
Rentoul, John, “Iraq death toll: a
response,” Independent, 10/18/13
- “New Iraq Death Toll Study,”
Independent, 10/17/13
Roberts, Les, Lafta, Riyahd, Garfield, Richard, Khudhairi,
Jamal, Burnham, Gilbert, “Mortality before and after the 2003 invasion of Iraq:
cluster sample survey,” The Lancet, 10/29/04
Russia Today, “’Our count is low’:
Behind the latest Iraq war-death study,” 10/18/13
Spagat, Michael, “Mainstreaming an Outlier: The Quest to
Corroborate the Second Lancet Survey of Mortality in Iraq,” Department of
Economics Department, University of London, February 2009
- private conversations, October 2013
Tirman, John, “War’s Violence and
Why It Matters,” Huffington Post, 10/17/13
Van Der Laan, Mark and de Winter, Leon, “Statistical
Illusionism,” U.C. Berkeley, 2006
Van Reet, Brian, “New Report Cites
Half-a-Million War Related Dead in Iraq,” Daily Beast, 10/23/13
12 comments:
Thanks for this, but I feel a bit quizzical about the 'duelling reports' formulation of much of the coverage (which you have adopted here as well). The Lancet and PLOS are journals, the authors of the various studies - I would like to hope - were all doing their best to approximate the casualty levels emerging from various sources being meted out to the Iraqi people. Some may have allowed bias to enter their methodologies. None can claim authority on findings - and I believe this is why the authors of the study published in the Lancet can believe that these findings are well within the range ascribed by his own study ... because they are.
All such studies are approximations and while the studies published in the Lancet estimated on the high end and others on the low, the methodological arguments seem to have been captured as proxies for the 'pro-war' and 'anti-war' simplistic formulations of the 2002-03 period in English-language media. This is unfortunate in the extreme IMO. This isn't a critique of this site or of your writing - your work here over the past decade has been superlative.
I'm just wondering if the competitive formulation of study-vs-study, estimate-vs-estimate, doesn't inure readers to the tragedy and the human suffering. To be crass: 'oh, its *only* 400K and not a million excess deaths? Well then ...'
I believe this framing emerged from that pre-war Anglo-American debate ... but, it fails to *once again* take an Iraqi perspective. That we can detail each American, British or Mongolian death in great detail and with a modicum of respect for their humanity ... we still discuss 'ranges' for Iraqis and argue over the figures to be accepted. For all its supposed methodological basis, such a framing of the discussion fails to acknowledge the victims as human beings.
Thanks for your efforts here to inform Americans looking for an alternative to mainstream media ... but, the framing will continue to confound the discussion going forward.
The problems with the two Lancet reports are huge including oversight of the survey teams, whether they were random in their selection of houses, whether they had a mix of low and high end violent neighborhoods, their confidence intervals, etc. It goes way beyond just coming up with the high death estimates.
Here's two articles that cover most of the critiques of the Lancet along with an interview I did about them:
http://musingsoniraq.blogspot.com/2012/08/the-major-flaws-with-lancet-reports-on.html
http://musingsoniraq.blogspot.com/2012/08/major-flaws-with-lancet-reports-on.html
http://www.musingsoniraq.blogspot.com/2013/07/a-critique-of-lancet-reports-on-iraqi.html
You're previous posts were very well-written and informative.
The issue depends on what you mean by 'problems'.
To my mind those conducting all these surveys and pieces of research used the methods they deemed "best practices" given the time and other constraints they were operating under [assuming that all were well-intended as I have *no* evidence to conclude that they were inherently biased].
To my mind, aside from the 'inside baseball' debates over methodologies, the sole difference of all these studies is the 'take away' or 'headline' number that became associated with there release. The Lancet-published studies drew attention as they were 'high'? They had a range and asserted a figure within that range ... a number that the media (and activists) picked up on and ran with. The confidence interval was varied, the range was varied, but all the studies were within a certain scope and not really at odds scientifically ... they *were* at odds due to the assumptions involved, not the maths.
So, my question is are we attacking the studies (and their authors) themselves - or the perception they created? To my mind it is really the later ... and if so, this is an inherently *political* discussion.
Why? Because we (whoever 'we' is) don't like the 1 million deaths number, but we're OK with the 500K number.
If I'm Iraqi, my reaction is: 'huh? what is the difference?'
There are a tremendous number of casualties here. The humanitarian plight and horror show of what Iraqis have been exposed to over the last ~three decades at the hands of the Ba'thist regime and the Anglo-American 'international community' is not mitigated by there being only 500K casualties. I'm assuming that is not what you are arguing.
Those teams of researchers that conducted and published the Lancet findings? They did a job. At the time no one had any real idea of the humanitarian impacts; they pegged a number and came under immense scrutiny. Unless there is evidence that they erred for political expediency I don't see the point in hammering away on their findings.
What am I missing?
I think there were so many problems with the Lancet studies especially the 2006 one that I do not find them credible. The other studies done at the exact same time as the two Lancet's and afterward have all come up with much different figures. The Iraq Living Conditions and the Iraq Family Health Survey worked in Iraq at the exact same times as the two Lancet's, surveyed far more Iraqis, had quality control, better oversight of their survey teams, etc. Why should you let the two Lancet paper's off when others were able to do a much better job under the same circumstances?
As for the ranges, they were so wide on the two Lancet's that many don't find them credible. That's not just "inside baseball" about methodology, they question the veracity of the reports themselves.
The authors of the Lancet did have a political agenda as well. Both were released right before elections in the U.S. The critiques about them are mostly methodological, but there's also a political debate about them because the authors were political.
Finally, when you're talking about the total death estimates you're comparing different time periods. The new PLOS Study and Iraq Body Count all estimated that 100k died from violence from 2003-2011. That's also supported by the U.S. military numbers, and the morgue and Iraqi govt death certificate figures. The 2nd Lancet said around 600k died from violence from just 2003-2006. I think those differences do matter. You're talking 6-10 times difference. I think people should be striving for the most accurate figures they can when they're dealing with any country or issue. Would you say it doesn't matter if 100k die from cancer in the U.S. each year or 1 mil? Would it matter if the high school drop out rate in the U.S. is 10% or 50%? You seem to be arguing it's a lot and that's all that matters. How does that help explain anything? When people died, how they were killed and where all help give better understanding of the violence that has plagued Iraq since 2003. To me you need the best possible info to study those events.
jmeasor, your complaint here seems to be that the findings in this new study aren't really different from the Lancet studies of the past:
"this is why the authors of the study published in the Lancet can believe that these findings are well within the range ascribed by his own study ... because they are."
The problem is they aren't. They diverge dramatically, including on issues which the Lancet authors themselves emphatically insisted were correct for years, such as "600,000" killed directly by war violence in just the first three years of the war. The Lancet authors insisted this was correct, repeatedly and emphatically, and repeatedly attacked other sources such as Iraq Body Count and the other surveys that came to much lower conclusions about deaths from violence.
Now they appear to want to take the line that the Lancet survey and this new one are consistent specifically (and only) on "excess deaths", where cause of death no longer matters - killings by bullets or bombs are the same thing as cardiovascular conditions or cancer and it doesn't matter which one the studies claim they are - and consistency is defined apparently as a bare overlap at the edges of two enormous confidence intervals. This is just self-serving and hypocritical opportunism on their part.
To the PLoS authors' credit, they did provide some raw data for the survey along with their report, so we can look at the data and do a direct comparison to the Lancet report time period. It's difficult to draw exact estimates or exact intervals, as these depend on a number of interpretive choices, but you can see in the data that for the Lancet period (March 2003 - June 2006), the PLoS survey would correspond to something like 200,000 excess deaths. The interval on this would be wide, probably similarly wide as the full estimate or maybe wider, so you could posit an upper edge of the interval at something close to perhaps 400,000. The 2006 Lancet survey interval for excess deaths was "392,979 to 942,636". So what in the world could be meant by a claim that the new findings are "well within the range" here?
It's only possible that the two enormous CI's might barely overlap at the outer margins. But to claim that these studies show basically the same thing is ridiculous, and this is not the standard the Lancet authors themselves insisted upon for years. For example, in 2004 the Lancet study found a range of 8,000-194,000 excess deaths. The Iraq Body Count numbers solely for civilians killed in violence for that period were well within that interval. So did the Lancet authors say that their findings and IBC were consistent, that neither source disproved anything about the other, and that the difference in the central figures didn't matter because IBC fell well within their range? NO! They denounced IBC as a "gross underestimate" and cooked up bogus analyses to insist that IBC was low by "factors of five to ten" and that "the IBC monitoring network cannot be more than 20% complete", and instigated a sustained public smear campaign to discredit IBC. Much of the sordid details on this can be found here:
http://www.iraqbodycount.org/analysis/reference/pdf/a_defence_of_ibc.pdf
The Iraq Living Conditions survey was also well within that 8,000-194,000 range. So did the Lancet authors accept that there was no contradiction? Nope. They first tried to argue that the two were basically the same if you account for the fact that they're measuring different things, but eventually they just wound up insisting that the ILCS was a "gross underestimate".
But, ok. Now that it serves them to change their whole position in order to claim some kind of consistency between the Lancet and this new study, let's just accept this as the new standard. If anything overlaps at all, even barely, with the enormous intervals associated with these surveys, then they are showing the same thing. Fine.
(see following post)
(continued...)
So what about the most contentious claim of the 2006 Lancet study, that 600,000 were killed directly by violence? This was the most widely disputed claim about that study. Did the authors respond to that criticism by saying that, well, maybe it wasn't violence. It could be cardiovascular conditions or whatever. All that matters is that they are vaguely in some way "excess deaths". Again, no. They insisted emphatically that these were deaths from violence, and even insisted that it was probably an *underestimate*, and that all the other sources measuring much lower numbers from violence were *wrong*, had bad methodology and were serving as propaganda for war proponents.
So how does this new survey compare to the 2006 claim about violence? For that Lancet period the PLoS survey would suggest somewhere around 100,000 deaths from violence. The upper edge of the interval would be less than 200,000. Contrast to the Lancet claim: 601,027 (426,369 to 793,663).
The two surveys are not even remotely close. They are completely contradictory on the issue of violence. The PLoS survey (like the IFHS and other sources before it) suggests that the Lancet authors were wrong - hugely wrong - in all their claims about violence, claims which they emphatically insisted upon for years. Every substantive claim made about violence by the 2006 survey and its authors that differed substantially from other sources was wrong, wildly wrong.
Authors like Gilbert Burnham who was involved with both surveys should be more forthright about this, instead of trying to evade the whole issue with a false and opportunistic claim that there's no contradiction because the enormous CI's of the two surveys might barely overlap on "excess deaths".
I sympathize with your argument that debating the 'right' numbers doesn't matter as much as the general conclusion about the fate of Iraqis throughout this war, but the only relevant point or purpose to a study like this is to try to get more accurate with the numbers. If the point is just that a lot of Iraqis died, then anyone who's been paying any attention should already know that and nobody needs this study. But the methodological debate and the fight over numbers is a fight that has been instigated and picked primarily by the Lancet authors and their supporters. And they have continued for years to crank out all kinds of dubious propaganda to create the illusion of support for their side in their chosen fight. Some of it downright fraudulent, like this: http://www.ncbi.nlm.nih.gov/pubmed/23388622.
They see themselves in a methodological war, a war in which they "must prevail" over others who use different methods (see here: press.thelancet.com/mortality.pdf). But the fact is that in this new survey, many of their most contentious claims about the Iraq war have been refuted (again), and this time by one of their own surveys, and sadly they refuse to be honest about that.
joshd.ibc and Joel, my ‘complaint’ was perhaps off the mark for this venue … I did not see it as a complaint in the first instance, I saw it as an intervention to point out that I don’t believe that the “story” of the PLOS study was to refute the Lancet studies, but rather to try to approximate a benchmark number of Iraqi excess deaths. Moreover, that the story to take from the PLOS study being published is that that number is indeed quite large. I also made a larger point - that we are still at a point where we talk in terms of clinical distance from Iraqi deaths that keeps many from understanding the tragedy that has befallen people living within Iraq over that last three decades.
The focus here, on the occasion of the announced study by academics published in PLOS Medicine (October 2013), was that the current study refutes the studies published in the British medical journal the Lancet (October 2004 and October 2006). All three studies were peer reviewed (as was the Iraq Family Health Survey), while all the other estimates were not; peer review may not mean much to people - or it may. I believe what matters to people is the ‘take away’ number. The enormous efforts to detail problems with the studies published in the Lancet always stood out to me for one reason, that such efforts are normally almost universally limited to academic debates rather than becoming public spectacles in and of themselves; meaning that their public appearance speaks to the Lancet studies being provided a more rigorous public vetting due to their *political* importance rather than some simple academic argument. I would not argue that the political intentions of the Lancet studies placed them and their authors in the firing line of such a retort. Fair enough.
However, I would assert that everyone mentioned, indeed anyone looking to invest their time and limited resources in conducting such a study, cares deeply about finding an accurate number. I am not involved in the heated debate that ensued the publication(s) of the studies in the Lancet. If you were and feel personally engaged with the authors of the Lancet studies then I apologize if I appeared to be siding with them and supporting their study per se. I am not.
I am saying that the ‘take away’ most all readers would have is that the Lancet studies made a claim that ~600K-1,000K people were dead due to the Iraq war - seriously, no one will really remember the timeframe. Those same readers will remember that Iraq Body Count makes a claim that ~100K people were dead … again without refinement of time period, methods of arriving at that number, nor the fact that it was subsequently been updated. This is not to insult readers - its human nature and the fact that most people do not have the time nor the skills to invest in such debates. They trust that prestigious medical journals such as the Lancet and the public domain and ‘wisdom of crowds’ aspects of Iraq Body Count can allow them to act as sufficient approximations of the scope of the violence.
I think we can all agree that the current PLOS study by scholars from the University of Washington, Simon Fraser University, John’s Hopkins University and the Iraqi Ministry of Health is persuasive … or - if you prefer - more accurate AT WHAT IT IS CLAIMING TO DO. Indeed, Joel wrote: “While no survey can be authoritative the new PLOS one at least confirms that a few of the earlier estimates were capturing some of the death and destruction released by the Iraq War, while largely repudiating the Lancet articles.”
The detailed critiques Joel (and others) provided of the Lancet articles largely centre around expectations of what each study was to achieve, the assumptions implicit in their methodologies, and the differences in perception between the period(s) under study and the difference between ‘excess deaths’ and ‘people killed’. This study does considerably refine both methods and the findings. It finds ~500K excess deaths.
To my mind the Lancet and the PLOS studies are focussed on ‘public health’ rather than ‘body count(s)’. I should think it goes without saying that everyone involved in understanding the war on Iraq and Iraqis desires accurate figures. Virtually all political actors involved have incentives to either exaggerate or deflate these figures. Some actors can even desire the numbers to be in each direction (+/-) depending on the time period we might be examining and their potential for being “responsible” for the relative outcomes on the ground. For instance, U.S. forces see no good to come from ‘body counts’ - leading to their staunch efforts to not publicly acknowledge even the existence of such statistics. In the beginning they were dismissive of Iraq Body Count, public chastising any query based on the (then) current total deaths found on the site. Once the 2004 Lancet study emerged those same interlocutors would point questioners in the direction of Iraq Body Count in an effort to refute the Lancet accounts.
One could easily argue, along the same lines, that the current PLOS study clearly refutes the Iraq Body Count project. That, to my mind, would be unfair however. They are doing different things and have adopted methodologies according to what their designers want them to do.
Thus, my intervention was not to enter the fray and defend one study over another; it was in reaction to Joel’s post. Simply, what is the take away of the PLOS study? What is the story?
Is it what I was highlighting, that x-number of Iraqis were killed and that the enormity of that number should give us all pause. Or, is it that study x, y, or z has a more persuasive output and robust methodology to arrive at an acceptable approximation of those deaths?
As I said above, I would think the answer obviously depends on one’s position relative to the debate(s) surrounding this topic and the events. I’m asserting that were I an Iraqi I would think the take-away is *NOT* a debate about study x or y, but the now increasingly established large number of ~500K excess deaths, that the violence continues while we have this debate, and that it is increasingly insulting that we don’t actually name and vaporize those Iraqis who have been killed.
I appreciate you both being willing to engage on this topic. To my mind we need to not only get the grammar correct, but to think through how to best discuss this issue. Like the aftermath of Vietnam the way this conflict and its politics are commemorated and narrated will be a contentious matter moving forward. See the debate between the publication of David Petraeus’s piece “How We Won in Iraq” in Foreign Policy [http://www.foreignpolicy.com/articles/2013/10/29/david_petraeus_how_we_won_the_surge_in_iraq] and that of Wayne White in his rejoinder to Petraeus “On Iraq, Petraeus Still Marketing a Myth” [http://www.lobelog.com/on-iraq-petraeus-still-marketing-a-myth/]. Unlike the Vietnam conflict, this debate will be taking place in real-time and openly before a global audience. Joel’s post was read in Baghdad and Potsdam, Berkley and Little Rock, and everywhere in-between when it was posted.
"One could easily argue, along the same lines, that the current PLOS study clearly refutes the Iraq Body Count project. That, to my mind, would be unfair however. They are doing different things and have adopted methodologies according to what their designers want them to do."
I fail to see how you could credibly argue this, or along what lines.
I argued above that Joel was correct in his general assessment for specific reasons, citing the relevant statistics. The PLOS survey does indeed conflict sharply with the 2006 Lancet survey. The estimates for deaths from violence are not even remotely close to each other, either in the central figures or even in the wide ranges. Thus, they are in conflict, extremely so on violent deaths.
On "excess deaths" the best that could be said is that the two surveys might barely overlap at the outer margins of the huge 95% intervals.
Now how do you suppose it "refutes" Iraq Body Count? The only proper estimate they give in PLOS for direct deaths from violence is "132,000 (95% UI 89,000–174,000)", from their Sibling survey covering Iraqi adults 15-60. This is close to IBC for civilian deaths from violence, and IBC is certainly well within this range. It's actually *lower* than IBC if you do the more rigorous (in other words, correct) comparison and include combatants in IBC, like the survey does. In that case, the corresponding IBC number would be around 160,000 - higher than the PLOS central number, and again well within the range.
They also make a vague (and misleading imo) claim that "more than 60%" of excess deaths from their Household survey were from "violence". They do not publish any proper estimate however, just this vague proportion.
From this you could perhaps draw an estimate of around 200-250,000 violent deaths, which would then carry a huge range around it. This central figure would come out somewhat higher than any corresponding number from IBC, but IBC would still fall well within the 95% range.
So, no. There is nothing in the PLOS survey that "refutes" anything at all in IBC. Nothing. It does however clearly refute (again) the contentious claims about violent deaths made by the Lancet survey.
With statistics like this the 95% interval is generally accepted as the threshold for rejecting a hypothesis ("refutation"). So if your hypothesis is that "601,027 (426,369 to 793,663)" violent deaths occurred during 2003-2006, as claimed by the 2006 Lancet survey, the PLOS survey stands as a rejecting that hypothesis with 95% confidence. Contrarily, it refutes nothing at all about IBC with 95% confidence.
Even if you take the main PLOS estimate of "405,000 (95% UI 48,000–751,000)" for "excess deaths" and (wrongly) compare that to IBC, even this apples/oranges comparison doesn't refute anything from IBC. Everything from IBC either comes close to the PLOS central numbers, or falls well within their 95% ranges. That is not the case for the Lancet survey, which is why Joel's basic point is correct.
It is the case that both IBC and PLOS can be right. Nothing in either source conflicts with the other in any statistical sense.
It is, however, NOT the case that both PLOS and the Lancet survey can be right. At least one of them has to be very wrong.
The equivalence you're hoping to draw here is just wrong.
Einstein once had the words “not everything which can be measured counts, and not everything which counts can be measured.”
We are simply speaking past each other at this point and to my mind that is tremendously sad. I in no manner wished to challenge IBC … it is clear that you are assuming that I am somehow affiliated with the authors of the Lancet studies, defending them, or attacking IBC. I am not.
I am talking about ‘politics’ and the use of these metrics, how they have been and will be deployed politically. Perception(s) as opposed to the rigorous clarity you seem to feel is warranted by the debates above. Simply, the scope of those having been killed is becoming lost in the heat of such a debate, rather than bringing the light that we both profess to want to emerge.
My argument is:
a) a lot of Iraqis have died over the last two decades. That there seems to be (as yet) little recognition of this or lamentation for their loss in the Western media and political discourse. In such a light such heated debates over the figures can come across as crass. This does have implications - especially in our proverbial global village. Iraqis are reading what you write.
In the original post announcing the new study, nor in the subsequent discussion between the three of us, has my point even been nodded at. I’m not saying that every comment or discussion regarding Iraq from 2003- requires such a warrant. It would lose any notion of authenticity similar to how everyone acknowledges their support for “the troops”. However, with the publication of such a study I think is that Iraqis have suffered - not ‘Lancet authors refuted’. It would be like the Boston Globe running a banner headline last week such as “Red Sox win World Series, Brian Cashman is a failure”.
b) people in the main will only take from all such studies a ‘take away’ number. They do so for many reasons - good and bad. Lament it or criticize it if you wish, but that is the price of producing grist for the public mill.
Media figures, politicos, activists, and even analysts and researchers from outside the narrow domain focussed on this topic with adequate statistical numeracy will walk away saying “X”-number of Iraqis died. I believe you know this, and acknowledge it to be the case, as you put the figure prominently on the page at IBC.
Politics will dictate that these numbers will be used *politically*, thus this is not simply an argument over semantics and methods between researchers. As I suspect all participants are well aware that even if they themselves are as pure as the driven snow that their research will be utilized for such arguments. I would like to therefore assume that everyone involved is conducting their research with the best of intentions - but, acknowledge that their output will get dragged into the pro-war and anti-war debate. This is why I argue in ‘a’ above that there is a concern; that debate needs to now be surpassed with a recognition of the implications of the war. IBC isn’t diminished if its acknowledged that more Iraqis died than it count allows once one takes other modalities into account. IBC is not accounting for Iraqi children who will never be conceived due to the war, the Lancet and PLOS Medicine studies are. We can’t compare vegetables and meteorites.
As to the politics I believe the experience of IBC may be illustrative in this regard. In interviews with U.S. military and state officials prior to the Lancet publications a strict public line was adopted eschewing “body counts”. Moreover, as I was there and conducting interviews at the time, many derisive things were said off-the-record about the IBC project that reporters and activists had increasingly begun to quote from. Once the Lancet studies entered the public fray, with figures five- to ten-times greater, these same officials began to direct questioners to IBC (!). I’m pretty sure IBC didn’t change, but the political actors involved wanted to use its work to drive public perception(s). Thus, …
c) all of these projects are attempting to do different things.
I don’t believe you are making the claim that IBC is a scientific study. It may be novel, and perhaps can be seen as a new form of research based on the dynamic and sweeping changes in our media landscape taking place coterminously with the third Iraq war. I don’t necessarily think that peer reviewed and terribly sophisticated studies of the nature published in The Lancet or PLOS Medicine are necessarily any “better” … all these studies are making truth claims. Activists and the U.S. government are OK with exactly the same metrics when used to identify excess deaths in Darfur, for instance, but opposed to them when adopted in other settings. Its the nature of the beast in attempts to quantify human events.
IBC is claiming ~120K Iraqis have been killed - “non-combatants killed by military or paramilitary action”. Public health studies such as that found in the Lancet and PLOS Medicine are measuring ‘excess deaths’ - people who would be there were it not for intervening variables. Babies born, cancer patients who would have received care or not even contracted the disease, and especially scores of people who are exposed to water-born disease and pathogens. There is a place for all such data within our discussions of the impact of events. As a researcher I don’t think any such studies can be seen as gospel. Indeed, historians and social scientists are still debating the scope of acceptable figures for how many people were killed during the Second World War and many other events.
Readers will have greater or lesser confidence based on their political views (a bias most everyone enters the fray carrying) and comfort with the methodologies utilized (based on their technical knowledge, or lack thereof). Political actors looking to avoid accountability - not IBC or other researchers - will largely have ‘won’ their point if the debate gets too bogged down and confusing to the public. Clearly authors/researchers having conducted such efforts are engaged in a heated debate with one another. I think this debate is colouring and informing your reading of my comments above. If you wish to see the Lancet studies retracted from that prestigious journal, their authors publicly flayed and shamed, or for them to acknowledge wrongdoing, I believe you may have lost the tenor of the original goal: to measure the impact of the 2003 invasion and occupation.
Thus, I was not saying that IBC was in error. I was saying that most people will see the PLOS Medicine study as setting the death toll at ~500K and the IBC as remaining at ~120K and may therefore assume that the newer ‘study’ is more persuasive - even definitive. As explained above, readers who have examined each study will know that you are looking at somewhat different things. Does the PLOS Medicine study refute the Lancet studies? Certainly, but as they are both working within an environ of peer-reviewed research I believe the academic take away will be - “new study surpasses earlier study” which is how the scientific process is supposed to work is it not? If the Lancet authors want to argue with those findings they can conduct another study :-) Similarly, were future researchers to find a secret cache of government data with detailed death totals greatly in excess of that found on IBC it may refute the erstwhile efforts of IBC to curate public data. However, that would in no way diminish your efforts or speak to the ICB effort being somehow incorrect. Put another way - with the PLOS Medicine study now out and seemingly accepted, why even mention the Lancet study if it is so disreputable?
I applaud the efforts of IBC, as well as the studies published in the Lancet and PLOS Medicine. I don’t see how the debate between these researchers - no matter how serious any study’s methodological flaws may be - can take precedence over the tragedy that has befallen the Iraqi people.
Post a Comment