Debunking the Lancet Report

[Yirmeyahu]

Mortality Study

I apologize for an error. I was in a rush, not reading carefully, and presumed this thread was regarding the report published in the Lancet in October 2006. I replied after briefly scanning the original post. After re-reading it, I see that it deals entirely with the 2004 study.

The 2006 study estimated nearly 600,000 Iraqi deaths since the invasion. What I wrote above was regarding the 2006 report.

There are, however, a few relevant points that apparently need reiterating, as several points are true for both studies.

Can you tell me where these nearly 550K dead bodies are?

Again, it's not a question of counting bodies. IBC doesn't count bodies. It counts civilian deaths reported in the media by at least three sources. Obviously, most deaths aren't reported. IBCs figure is an extremely conservative estimate therefore.

When the John Hopkins researchers performed their research in 2006, in most cases, 9 out of 10 times, the household produced a death certificate.

I wouldn't state that it's bad practice, but their precision in their estimates is horrible, and some of their methodology doesn't hold up to the required assumptions.

Again, the 2006 study used the standard, well-accepted methodology. The 2004 used a similar methodology. UNICEF uses the same methodology to do studies of mortality rates. So does the U.S. government. As for the "required assumptions", I don't know what that's supposed to mean.

If we're going to discuss the matter, the 2006 study is more relevant, and improvements were made in the methodology, including an increased number of clusters studied.

For those who want to look at the 2006 report:

http://www.yirmeyahureview.com/lancet_mortality.pdf

[Stan187]

Hi, Yirmy. And just who the hell might YOU be?

A troll.

[Sombra]

Sombra,

The Lancet report can be discredited on its own terms. However, that doesn't necessarily discredit the notion that OIF has created excess deaths.

The problem is that most people don't understand the term "excess" deaths, and so too many arguments argue about apples and oranges. My thoughts are that OIF has created excess deaths by this point, with the unleashing of sectarian violence being the cause that has made the post war period more deadly. I'll post more later - got to go to work.

One of the main problems in the lancet report are two facts at least for me.

Reliable numbers of mortality before the war and the number of deaths certificates claimed.

Still, it is not only the people killed through direct violence which they are trying to count but the number of deaths due to the higher mortality. (This includes the old man dying of heat stroke becaue no electricity is avaible, children dying of dirty water etc, bombs, murders, military action.

Right now there are round about 100 violent deaths per day which are noted you dont know how many bodies you never find.

Up to now the US government doesnt even try to counter with numbers or a report on their own.

[Yirmeyahu]

Ad Hominum

A troll.

Hi Stan187.

You're welcome to attempt to dispute, in fact or logic, anything I've said. Ad hominum statements, you'll find, are unproductive.

[Stan187]

Hi Stan187.

You're welcome to attempt to dispute, in fact or logic, anything I've said. Ad hominum statements, you'll find, are unproductive.

Your "facts and logic" make you sound stupid enough, even without my help buddy.

[Yirmeyahu]

Like I said, the best estimate to date of Iraqi fatalities as a result of the 2003 US invasion is nearly 600,000. While many have tried to dismiss the results of this study, published in the Lancet medical journal, the methodology used is actually quite standard and used, for instance, by UNICEF to estimate mortality rates in countries.

[Shek]

I apologize for an error. I was in a rush, not reading carefully, and presumed this thread was regarding the report published in the Lancet in October 2006. I replied after briefly scanning the original post. After re-reading it, I see that it deals entirely with the 2004 study.

The 2006 study estimated nearly 600,000 Iraqi deaths since the invasion. What I wrote above was regarding the 2006 report.

There are, however, a few relevant points that apparently need reiterating, as several points are true for both studies.

The reports findings are about excess deaths - in other words, the findings depend upon the difference between pre-war mortality and post-war mortality. This will be very important a little bit later in the post.

Again, it's not a question of counting bodies. IBC doesn't count bodies. It counts civilian deaths reported in the media by at least three sources. Obviously, most deaths aren't reported. IBCs figure is an extremely conservative estimate therefore.

IBC counts the counts of people who report counted bodies. A distinction irrelevant to my point. Approximately 50K deaths have been reported by IBC. In order for the Roberts et al figure to be correct, then only one out of every ten violent deaths is reported, whether it is reported immediately after the incident, or whether it is reported as bodies that show up at the morgue.

Your contention that 9 violent deaths out of every 10 violent deaths goes completely unreported? Please explain to me how this is obvious. I would agree that IBC will under report deaths, but given the media victory that the insurgents and terrorists get by every death, and the readily available forms of communication (internet, cell phone), there is absolutely no way that IBC under reports by a factor of ten.

When the John Hopkins researchers performed their research in 2006, in most cases, 9 out of 10 times, the household produced a death certificate.

Great for them. Irrelevant to the arguments I have about the weaknesses of the Roberts et al study.

Again, the 2006 study used the standard, well-accepted methodology. The 2004 used a similar methodology. UNICEF uses the same methodology to do studies of mortality rates. So does the U.S. government. As for the "required assumptions", I don't know what that's supposed to mean.

This is a strawman. I haven't criticized, and have never criticized the underlying methodology. It is poor application of this methodology that sinks their findings.

As far as required assumptions, the third stage of the sampling is required to be random in order to use the underlying probability distributions to create your confidence interval. However, the study protocol for this stage follows:

The interview team were given the responsibility and authority to change to an alternate location if they perceived the level of insecurity or risk to be unacceptable.

Yet, there is no discussion of how many times this was invoked, and for what clusters. It is worth noting that they didn't explicitly state that this didn't happen. If you introduce bias into your sampling, then you bias the results and erode the validity of whatever your findings are.

So, are they hiding something, or are they slopply academics?

If we're going to discuss the matter, the 2006 study is more relevant, and improvements were made in the methodology, including an increased number of clusters studied.

Increased clusters. Woopty do! 33 to 47, a relative increase of under 50%. Unfortunately, in absolute terms, it's still terribly small, which is why the precision of their finding sucks!

To put this into perspective, the UNDP study completed in 2004, just a few short months before the 2004 October surprise by Roberts et al, had 2,200 clusters, or 4,400% more clusters. Nice.

So, let’s look at a few other issues. I’ve mentioned benchmarking as a huge flaw. Roberts et al 2004 finds that the pre-war crude mortality rate to be 5.1, while their 2006 study finds that this rate was 5.5. However, neither of these CMR are close to the available benchmark, which is 8.5. By underestimating pre-war deaths, they nearly double the rate of estimated post war deaths. However, if you would like to suggest other benchmarks for the prewar CMR in Iraq, I’d love to see them.

Next, let’s look at the increase in excess deaths between the two studies. Excess violent deaths in the first study was approximately 48K. Twenty-two months later, this number had increased to approximately 601K. So, if we were to average this out, you have over 25K Iraqis killed by violence every single month, or nearly 1000 a day. However, the UNAMI has reported that the deadliest month for civilians was October, with 3,709 civilians being killed. So, based on the deadliest month during the entire two year period, the figures still end up being 21,000 short. Where are these 21,000 dead bodies buried?

[Yirmeyahu]

The reports findings are about excess deaths - in other words, the findings depend upon the difference between pre-war mortality and post-war mortality.

Yes. That goes without saying.

Your contention that 9 violent deaths out of every 10 violent deaths goes completely unreported?

Unreported by at least two major independent online English-language media sources, yes. IBC also excludes "excess civilian deaths caused by criminal action resulting from the breakdown in law and order which followed the coalition invasion," unlike the John Hopkins study.

Why, therefore, it is incredible to you that IBC should underreport the actual figure by a factor of only 10 (actually, closer to 12), I know not. The nature of the IBC methodology predertimines that the IBC estimate is an absolute bare minimum figure.

The John Hopkins study, unlike IBC, used a scientific methodology for arriving at mortality rate estimates. In fact, it's a well-accepted methodology, used by UNICEF, among others, to estimate mortality rates.

This is a strawman. I haven't criticized, and have never criticized the underlying methodology.

You said:

...their precision in their estimates is horrible, and some of their methodology doesn't hold up to the required assumptions.

That is a criticism of their methodology. It was to that remark that I responded by saying, "Again, the 2006 study used the standard, well-accepted methodology", to which you responded that this was a "straw-man" as you'd never criticized their methodology.

However, the study protocol for this stage follows:

Quote:
The interview team were given the responsibility and authority to change to an alternate location if they perceived the level of insecurity or risk to be unacceptable.

Sure, another randomly selected location. This does not introduce "bias", but it is a reasonable way to help ensure team safety. But you presume a) this occurred, and b) further, that by choosing secondary locations the study was so fatally flawed as to be dismissable. Your argument rests on too many presumptions and not enough facts. One could just as easily assume this was never invoked.

Increased clusters. Woopty do! 33 to 47, a relative increase of under 50%. Unfortunately, in absolute terms, it's still terribly small, which is why the precision of their finding sucks!

An increase of almost 50%. That's hardly an insignificant increase. Neither is it "terribly small". The US used a similar number of clusters in a study of Kosovo.

The precision could have been increased, yes, with more clusters, but as is, the sampling was large enough to be 95 percent certain that the number of deaths is between 400-900 thousand.

Let's assume its as few as 400,000 deaths. The chances of the number being that low are small. But even if it were, that's still a significant number of deaths as a result of the invasion.

Despite flaws, the study tells us that we can be quite certain that the number of deaths is more than 400,000.

To put this into perspective, the UNDP study completed in 2004, just a few short months before the 2004 October surprise by Roberts et al, had 2,200 clusters, or 4,400% more clusters. Nice.

The clusters from UNDP study you refer to included 10 interview each. Each cluster from the John Hopkins study included 40 households; data from 1,849 households containing 12,801 individuals.

Roberts et al 2004 finds that the pre-war crude mortality rate to be 5.1, while their 2006 study finds that this rate was 5.5.

Yes, their secondary findings confirm the first. Even though different households were sampled, the number they arrived at each time were fairly close.

The CIA gave a death rate of 5.84/1000 for 2003.
http://permanent.access.gpo.gov/lps35389/2003/iz.html

The US Census Bureau Global Population Profile gave a rate of 6/1000 for 2002.
http://www.census.gov/prod/2004pubs/wp-02.pdf

The 2004 UN figures are actually based on a 1997 census and based on projected trends from that year.
http://www.un.org/esa/population/pub...l/Chapter7.pdf

[Sombra]

Excess violent deaths in the first study was approximately 48K. Twenty-two months later, this number had increased to approximately 601K. So, if we were to average this out, you have over 25K Iraqis killed by violence every single month, or nearly 1000 a day

in other words, the findings depend upon the difference between pre-war mortality and post-war mortality. This will be very important a little bit later in the post.

Shek these are two of your own statements. Can you please point out were the lance reports says that 600k dead are only due to violent deaths? As far as I was reading it the were comparing mortality rates. Wiping out an infrastructure for example will kill more peoople with unclean water than bullets. Still its a direct consequence of the conflict.

I share your concern what is the benchmark figure (mortality rate) before the war compared to the number now.

[Shek]

Shek these are two of your own statements. Can you please point out were the lance reports says that 600k dead are only due to violent deaths? As far as I was reading it the were comparing mortality rates. Wiping out an infrastructure for example will kill more peoople with unclean water than bullets. Still its a direct consequence of the conflict.

I share your concern what is the benchmark figure (mortality rate) before the war compared to the number now.

Sombra,

I've attached the full Roberts et al report to this post. If you look on page 2 of the report under key findings, you'll find the following:

We estimate that through July 2006, there have been 654,965 “excess deaths”—fatalities above the pre-invasion death rate—in Iraq as a consequence of the war. Of post-invasion deaths, 601,027 were due to violent causes.

As far as the benchmarking figure, I've seen people try to use the CIA figure before to defend Roberts et al. However, if you find the CIA numbers to be credible, then you end up having to explain why you believe their pre-war figure but not their 2006 figure:

https://cia.gov/cia//publications/factbook/geos/iz.html
5.37 deaths/1,000 population (2006 est.)

Unless one can explicitly demonstrate a change in methodology, then either both figures are correct or neither is correct.

Last night, I was finally able to determine the source of the CIA and US Census figures when I was looking at the US Census link that Yirmey provided, and they are one and the same. The figures are taken from the International Data Base (IDB).

http://www.census.gov/ipc/www/idbsprd.html

So, we are left with the UNDP benchmark, and if you don't like that one, then we could use the World Bank's benchmark as reported by the WHO:

http://www.emro.who.int/iraq/pdf/Hea...emsProfile.pdf

1990 crude death rate 9.18
1995 crude death rate 10.44
2000 crude death rate 8.84
2002 crude death rate 7.80

If you look at the pre-war death rates for both the 2004 and 2006 Roberts et al studies, you'll see that both benchmark rates are outside of the 95% CI for their finding.

Also, while we are on the topic of benchmarking, the three examples that Roberts et al use at the end of their 2006 study are poorly motivated, i.e. they don't provide any reasoning as why East Timor, Vietnam, and DRC are analogous to Iraq, and therefore provide a logical benchmark. Furthermore, their figures for East Timor rely on a secondary source that is double of what a scientific study found (and turns up quite easily when you search for it). Using a secondary source is 1) poor academics, and 2) given how easy it was for me to find the study, I question as to why they would choose not to use it and instead use an estimate with no basis in methodology.

[Shek]

More Roberts et al criticism . . .

Could 650,000 Iraqis really have died because of the invasion?-News-World-Iraq-TimesOnline

From The TimesMarch 05, 2007

Could 650,000 Iraqis really have died because of the invasion?Anjana Ahuja
The statistics made headlines all over the world when they were published in The Lancet in October last year. More than 650,000 Iraqis – one in 40 of the population – had died as a result of the American-led invasion in 2003. The vast majority of these “excess” deaths (deaths over and above what would have been expected in the absence of the occupation) were violent. The victims, both civilians and combatants, had fallen prey to airstrikes, car bombs and gunfire.

Body counts in conflict zones are assumed to be ballpark – hospitals, record offices and mortuaries rarely operate smoothly in war – but this was ten times any other estimate. Iraq Body Count, an antiwar web-based charity that monitors news sources, put the civilian death toll for the same period at just under 50,000, broadly similar to that estimated by the United Nations Development Agency.

The implication of the Lancet study, which involved Iraqi doctors knocking on doors and asking residents about recent deaths in the household, was that Iraqis were being killed on an horrific scale. The controversy has deepened rather than evaporated. Several academics have tried to find out how the Lancet study was conducted; none regards their queries as having been addressed satisfactorily. Researchers contacted by The Times talk of unreturned e-mails or phone calls, or of being sent information that raises fresh doubts.

Iraq Body Count says there is “considerable cause for scepticism” and has complained that its figures had been misleadingly cited in the The Lancet as supporting evidence.

One critic is Professor Michael Spagat, a statistician from Royal Holloway College, University of London. He and colleagues at Oxford University point to the possibility of “main street bias” – that people living near major thoroughfares are more at risk from car bombs and other urban menaces. Thus, the figures arrived at were likely to exceed the true number. The Lancet study authors initially told The Times that “there was no main street bias” and later amended their reply to “no evidence of a main street bias”.

Professor Spagat says the Lancet paper contains misrepresentations of mortality figures suggested by other organisations, an inaccurate graph, the use of the word “casualties” to mean deaths rather than deaths plus injuries, and the perplexing finding that child deaths have fallen. Using the “three-to-one rule” – the idea that for every death, there are three injuries – there should be close to two million Iraqis seeking hospital treatment, which does not tally with hospital reports.

“The authors ignore contrary evidence, cherry-pick and manipulate supporting evidence and evade inconvenient questions,” contends Professor Spagat, who believes the paper was poorly reviewed. “They published a sampling methodology that can overestimate deaths by a wide margin but respond to criticism by claiming that they did not actually follow the procedures that they stated.” The paper had “no scientific standing”. Did he rule out the possibility of fraud? “No.”

If you factor in politics, the heat increases. One of The Lancet authors, Dr Les Roberts, campaigned for a Democrat seat in the US House of Representatives and has spoken out against the war. Dr Richard Horton, editor of the The Lancet is also antiwar. He says: “I believe this paper was very thoroughly reviewed. Every piece of work we publish is criticised – and quite rightly too. No research is perfect. The best we can do is make sure we have as open, transparent and honest a debate as we can. Then we'll get as close to the truth as possible. That is why I was so disappointed many politicians rejected the findings of this paper before really thinking through the issues.”

Knocking on doors in a war zone can be a deadly thing to do. But active surveillance – going out and measuring something – is regarded as a necessary corrective to passive surveillance, which relies on reports of deaths (and, therefore, usually produces an underestimate).

Iraq Body Count relies on passive surveillance, counting civilian deaths from at least two independent reports from recognised newsgathering agencies and leading English-language newspapers ( The Times is included). So Professor Gilbert Burnham, Dr Les Roberts and Dr Shannon Doocy at the Centre for International Emergency, Disaster and Refugee Studies, Johns Hopkins Bloomberg School of Public Health, Maryland, decided to work through Iraqi doctors, who speak the language and know the territory.

They drafted in Professor Riyadh Lafta, at Al Mustansiriya University in Baghdad, as a co-author of the Lancet paper. Professor Lafta supervised eight doctors in 47 different towns across the country. In each town, says the paper, a main street was randomly selected, and a residential street crossing that main street was picked at random.

The doctors knocked on doors and asked residents how many people in that household had died. A person needed to have been living at that address for three months before a death for it to be included. It was deemed too risky to ask if the dead person was a combatant or civilian, but they did ask to see death certificates. More than nine out of ten interviewees, the Lancet paper claims, were able to produce death certificates. Out of 1,849 households contacted, only 15 refused to participate. From this survey, the epidemiologists estimated the number of Iraqis who died after the invasion as somewhere between 393,000 and 943,000. The headline figure became 650,000, of which 601,000 were violent deaths. Even the lowest figure would have raised eyebrows.

Dr Richard Garfield, an American academic who had collaborated with the authors on an earlier study, declined to join this one because he did not think that the risk to the interviewers was justifiable. Together with Professor Hans Rosling and Dr Johan Von Schreeb at the Karolinska Institute in Stockholm, Dr Garfield wrote to The Lancet to insist there must be a “substantial reporting error” because Burnham et al suggest that child deaths had dropped by two thirds since the invasion. The idea that war prevents children dying, Dr Garfield implies, points to something amiss.

Professor Burnham told The Times in an e-mail that he had “full confidence in Professor Lafta and full faith in his interviewers”, although he did not directly address the drop in child mortality. Dr Garfield also queries the high availability of death certificates. Why, he asks, did the team not simply approach whoever was issuing them to estimate mortality, instead of sending interviewers into a war zone?

Professor Rosling told The Times that interviewees may have reported family members as dead to conceal the fact that relatives were in hiding, had fled the country, or had joined the police or militia. Young men can also be associated with several households (as a son, a husband or brother), so the same death might have been reported several times.

Professor Rosling says that, despite e-mails, “the authors haven’t provided us with the information needed to validate what they did”. He would like to see a live blog set up for the authors and their critics so that the matter can be clarified.

Another critic is Dr Madelyn Hsaio-Rei Hicks, of the Institute of Psychiatry in London, who specialises in surveying communities in conflict. In her letter to The Lancet, she pointed out that it was unfeasible for the Iraqi interviewing team to have covered 40 households in a day, as claimed. She wrote: “Assuming continuous interviewing for ten hours despite 55C heat, this allows 15 minutes per interview, including walking between households, obtaining informed consent and death certificates.”

Does she think the interviews were done at all? Dr Hicks responds: “I’m sure some interviews have been done but until they can prove it I don’t see how they could have done the study in the way they describe.”

Professor Burnham says the doctors worked in pairs and that interviews “took about 20 minutes”. The journal Nature, however, alleged last week that one of the Iraqi interviewers contradicts this. Dr Hicks says: : “I have started to suspect that they [the American researchers] don’t actually know what the interviewing team did. The fact that they can’t rattle off basic information suggests they either don’t know or they don’t care.”

And the corpses? Professor Burnham says that, according to reports, mortuaries and cemeteries have run out of space. He says that the Iraqi team has asked for data to remain confidential because of “possible risks” to both interviewers and interviewees.

[Shek]

Here's the latest debunking article, NATIONAL JOURNAL: Data Bomb (01/04/2008).

Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros's Open Society Institute.

[S2]

A "sticky" thread?

Could we prominently place this somewhere for easy reference when assorted sock-puppets emerge?

[Shek]

Could we prominently place this somewhere for easy reference when assorted sock-puppets emerge?

Done. I suspect it will act more like honey than a fly swatter, but we'll see ;)

[ArmchairGeneral]

We estimate that through July 2006, there have been 654,965 “excess deaths”—fatalities above the pre-invasion death rate—in Iraq as a consequence of the war. Of post-invasion deaths, 601,027 were due to violent causes.

Do these people not understand the concepts of precision and significant figures?