Comparing Surveys of Iraqi Deaths During The War

2023 marked the twenty year anniversary of the invasion of Iraq. Some articles mentioned the possible Iraqi death toll using sources such as the 2007 Opinion Research Business survey that estimated over one million killed. Professor Michael Spagat of the Royal Holloway College has been a longtime observer of surveys on Iraqi casualties and released a paper comparing the major reports on the topic. Out of six surveys and the Iraq Body Count he found that four of them closely followed each other while three were outliers and not credible.

 

The 2004 Lancet study included 983 households across 33 clusters. The survey was all done along primary roads in Iraq and did not include the provinces of Basra, Qadisiya, Najaf, Salahaddin, Irbil, Dohuk or Muthanna. Because of those two factors Spagat did not believe it was representative of the country as a whole. All of those governorates besides Salahaddin had low levels of violence which would have greatly affected the estimated death figures. It should also be added that such a small number of clusters cannot adequately cover a nation. A 2008 paper argued that The Lancet was biased towards higher numbers because it questioned people along main streets where violence was more likely. This was reinforced by the fact that the surveys were likely conducted along primary roads and highways. The survey estimated 260,000 violent deaths with a 95% uncertainty interval of 65,000 to 640,000. Spagat wrote that such a huge range of possible deaths makes the findings meaningless. What does it mean if it believed there were 260,000 fatalities but that could be off by 200,000-400,000?

 

The 2005 Iraq Living Conditions Survey interviewed 21,668 households in 2,200 clusters across the country. The areas questioned covered the major urban areas making it largely representative of the country. It estimated 24,000 war related deaths with a 95% uncertainty interval ranging from 18,000 to 29,000 from March 2003 to May 2004.

 

The second Lancet survey released in 2006 has been thoroughly critiqued as unreliable. It was formally censored in 2009 by the American Association for Public Opinion Research for refusing to disclose basic facts about its methodology. The president of the Association criticized the Lancet authors for making public statements and arguments based upon their research while refusing to answer questions about how they conducted their work. In 2010 Spagat wrote a paper presenting evidence of data fabrication and ethical violations by the authors. One of them was suspended by his university Johns Hopkins for collecting identity information about people questioned despite promising not to do so. This pointed to flaws in how it did its research. The second Lancet interviewed 1849 households in 47 clusters covering March 2003 to June 2006. One cluster however reported deaths from July 2006 and was included anyway. Again like the first Lancet the minimal number of clusters can lead to misleading data because it doesn’t cover the entire country. It estimated 670,000 deaths with a 95% uncertainty interval of 440,000 to 940,000.

 

The 2007 Opinion Research Business survey interviewed 1499 people in an undisclosed number of clusters. It estimated 1,229,500 deaths from March 2003 to August 2007 with a range of 733,158 to 1,446,063. In January 2008 it released new figures based upon further surveying in rural areas. The new estimated was 1,033,000 deaths with a range of 946,000 to 1,120,000. This was based upon 2,163 interviews over 112 clusters. There was a major difference between the two surveys. The first asked about deaths of family members while the second one asked about deaths in households. The latter is a much narrower category involving people that live together. Families can include aunts, uncles, cousins, etc. who do not live in the same home. The number of deaths increased by a factor of five in southern provinces between the surveys as well as an increase in the central governorates. Spagat didn’t believe these reports were credible as a result.

 

The 2008 Iraq Family Health Survey interviewed 9,345 households over 971 clusters covering March 2003 to June 2006. It estimated 151,000 violent deaths with a 95% uncertainty interval from 104,000 to 223,000. This study covers the same period as the 2^nd Lancet except for the 2006 anomaly. The bottom of the 95% uncertainty interval for the 2^nd Lancet is nearly twice as high as the top 95% uncertainty interval for the Iraq Family Health Survey. That means one of them must badly mismeasure the number of deaths. Spagat believed that was the 2^nd Lancet study because of the various problems and critiques of it. At the same time Spagat found some issues with the Iraq Family Health Survey. One it increased its estimated by a factor of 1.6 because it said deaths were likely underreported by households. It also started by using a stable population for Iraq but then changed to a population growth rate to calculate its numbers. Finally it did not include some clusters because of security concerns and claimed some of the ones in Baghdad were four times as violent as completed clusters. These all led to an increased in the estimated fatalities.

 

The 2013 University Collaborative Iraq Mortality Study interviewed 1,976 households across 100 clusters covering March 2003 to June 2011. It provided three estimates of 200,000 deaths, 140,000 and 140,000. It had a 95% uncertainty intervals of 140,000 to 270,000, 100,000 to 180,000 and 110,000 to 180,000. The central estimate was 132,000 dead with a 95% uncertainty interval of 89,000 to 174,000.

 

Spagat then made graphs and tables to compare the six surveys and the Iraq Body Count and found similarities and major differences. The 2005 Iraq Living Conditions Survey, the 2008 Iraq Family Health Survey, the 2013 University Collaborative Iraq Mortality Study and the Iraq Body Count were all aligned with each other. None of them have the major problems as the two Lancet reports or the 2007 Opinion Research Business survey. Those three were far out of range of the other ones raising questions about their believability.

 

Tables on the central numbers of the different reports came to the same conclusion. The Iraq Living Conditions Survey’s central number estimate was 24,000 deaths. The Iraq Family Health Survey was 25,000. The Iraq Body Count was 28,000. The University Collaborative Iraq Mortality Study was 32,000. Those were all within the same range of 24,000 to 32,000 deaths. The 2^nd Lancet’s central number was 100,000.

 

Report	Central Figure for Deaths
Iraq Living Conditions	24,000
Iraq Family Health Survey	25,000
Iraq Body Count	28,000
University Collaborative Iraq Mortality Study	32,000
2nd Lancet	100,000

 

Spagat then broke down the central numbers for the surveys over different time periods and found the Lancets and Opinion Research Business reports were far outside of the others.

 

For March 2003 to September 2004 the University Collaborative Iraq Mortality Study had a central figure of 30,000 deaths by household and the Iraq Body Count had 35,000. The 2^nd Lancet’s was 190,000 and the 1^st Lancet was 260,000. 

 

The same difference was found from March 2003 to August 2007 with the University Collaborative Iraq Mortality Study having a central figure of 100,000 deaths and the Iraq Body Count with 130,000. The Opinion Research Business on the other hand had a figure of 1,000,000.  

March 2003-September 2004

Report	Central Figure for Deaths
University Collaborative Iraq Mortality Study (households)	30,000
Iraq Body Count	35,000
2nd Lancet	190,000
1st Lancet	260,000

 

March 2003-May 2005

Report	Central Figure for Deaths
University Collaborative Iraq Mortality Study (households)	35,000
Iraq Family Health Survey	48,000
Iraq Body Count	48,000
2nd Lancet	310,000

 

March 2003-Mid-2006

Report	Central Figure for Deaths
University Collaborative Iraq Mortality Study (households)	62,000
Iraq Family Health Survey	73,000
Iraq Body Count	77,000
2nd Lancet	670,000

 

March 2003-August 2007

Report	Central Figure for Deaths
University Collaborative Iraq Mortality Study (households)	100,000
Iraq Body Count	130,000
Opinion Research Business	1,000,000

 

Spagat’s conclusion was that due to the problems with methodology and how their figures compared to the others the first and second Lancets and the Opinion Research Business survey were not credible sources on Iraqi deaths. The other three reports and the Iraq Body Count aligned with each other and probably capture what the actual number of Iraqi deaths were resulting from the invasion and occupation of the country. This is very important because the 2^nd Lancet and the Opinion Research Business surveys are mentioned in the media much more than the other estimates largely because of their high figures. The critiques of their figures are largely known only to academics and researchers and not the general public.

 

SOURCES

 

Spagat, Michael, “The Violent Death Toll from the Iraq War: 2003-2023,” 7/27/23