In a speech last week to the newly formed Health Council of Canada, Roy Romanow urged an examination of the factors behind Canadians' health problems, such as air pollution.
It us [sic] understandable that he connects air quality with health. For years, we have heard from activists, academics and health boards that thousands of people drop dead each year from air pollution. In 2000 the Ontario Medical Association estimated that more than 1,900 Ontarians die annually from air pollution, making it a "public health crisis." The same year, the Toronto Board of Health attributed 1,000 deaths in the city to bad air quality. Last year, Senator Colin Kenny wrote in The Hill Times that air pollution from cars kills twice as many people as traffic accidents. The David Suzuki Foundation claims that 16,000 deaths -- about one out of every 15 -- in Canada are attributable to air pollution. In the U.S., the Environmental Protection Agency puts out comparable numbers, scaled up ten-fold, to estimate the death toll from air pollution.
These numbers in turn motivate ever-tightening restrictions on air emissions, including, here in Ontario, a precipitous scheme to shut down the province's coal-fired plants. When pressed for a reason why such an ill-advised policy would be pursued, at a time when Ontarians already face a serious and growing shortage of generating capacity, the answer always comes back: Because air pollution is killing thousands of people.
Since blackouts and high energy prices can, themselves, threaten public health, the cure might just be worse than the disease, especially if the mortality threat of air pollution has been overstated.
There is reason to believe it has been. Air pollution in Toronto is much lower today that it was in the mid-1960s. Sulphur dioxide concentrations measured at an air monitoring station at the corner of Bay and Wellesley fell by 95% between 1965 and 1997. Total suspended particulate levels fell by three-quarters over the same period. Even ozone levels trended downward over that interval. If pollution is killing thousands of people today, back in 1965 there would have been corpses all down the street.
So if the new Health Council really wants to make itself useful, it should indeed revisit the question of air quality and health, to figure out what is really going on. And it will find no better place to start than with a new, peer-reviewed paper just published in the respected Journal of Environmental Economics and Management. The authors, economist Gary Koop and environmental scientist Lise Tole, are both at the University of Leicester in the U.K., but luckily for us, they used data from Toronto. The title of their paper is the very model of British understatement: "Measuring the Health Effects of Air Pollution: To What Extent Can We Really Say that People are Dying from Bad Air?" If I'd found the results they got, I'd have chosen a title like: The Death Rate from Air Pollution in Toronto is a Big Fat Zero.
That, in a nutshell, is what they found. But in reporting it they are going up against a large industry of epidemiologists and environmental scientists who have for years asserted that air pollution significantly increases mortality in industrial cities like Toronto. So Koop and Tole had to explain not only why their conclusions fit the data, but also why others so easily, frequently -- and incorrectly -- come to the opposite conclusion.
The key problem: Many researchers report results from their statistical models without properly accounting for the uncertainty in the specification of the model itself. To address the uncertainty issue, Koop and Tole applied a technique called Bayesian Model Averaging. Understanding how it works is important to understanding the power of their argument.
Regression analysis involves taking a variable -- in this case the number of people who die each day in Toronto -- and explaining observed changes in it as a weighted sum of changes in other, independent variables thought to be causal factors -- in this case types of air pollution. The weighting factors are called regression coefficients. Regression allows a researcher to quantify the coefficients between, say, the number of daily deaths and the daily average ozone levels.
But factors that drive both variables may create an illusory correlation. For instance, a winter cold snap might cause power plants to ramp up production (increasing pollution) and also cause a flu outbreak that kills a dozen people. In that case, to avoid attributing the deaths to the pollution, the researcher has to control for the effect of the weather. Regression analysis allows a researcher to control for multiple potential independent factors and thereby isolate the relevant information.
Regression analysis also yields an estimate of the margin of error around the coefficients. If this margin is so large as to include the possibility that the coefficient is zero, the relationship is said to be insignificant.
Sometimes a coefficient is only significant when a particular list of independent factors is tested. So a researcher should try a variety of combinations, including time lags, trends and so forth. If many variables are involved, researchers could spend forever trying out different combinations. So most of the time they only try a few. This, however, may lead to data mining. Journals are more likely to publish significant results. This may create a selection bias if you only hear about the models that yielded significant results.
Bayesian model averaging tackles this problem by considering every feasible combination, and weighting the results of each one according to how well the model fits the data. The researcher then reports the probability-weighted results from all the combinations.
In most of the studies of air pollution and mortality, the researcher tries out only one or a few combinations. In the case of the Koop and Tole paper, they went a bit further, computing just over 567 trillion combinations -- and even that required some simplifying assumptions to get the number that low. Since they chose Toronto data to study -- daily mortality rates, daily air pollution levels and daily weather conditions over the period 1992 to 1997 -- they directly, and exhaustively, tested the claims being put forward by the Canadian air pollution alarmists. Armed with their results, they were able to state conclusions that will not easily be challenged.
They conclude that "the most probable model includes only weather variables." They find no significant effects of air pollution on mortality. The regression coefficients are very small and the model uncertainties are so large as to make the air pollution effects statistically indistinguishable from zero. This points to the need to beware of studies that only report on a small number of specifications. When Koop and Tole account for possible interaction effects (i.e. maybe pollution only matters in hot weather) they still find nothing but a Table "composed of zeroes (to three decimal places)." Being cautious academics, they take pains to explain that they are not dismissing any possibility of a relationship between air pollution and health, only that "our results indicate that there is no reliable statistical evidence for a link between air pollution and mortality" in the particular data set they studied. They also caution that "if, for no other reason than the adequate design of air quality standards that carry immense economic costs, it is important that researchers use appropriate statistical methods to estimate air pollution impacts."
Indeed. And we should be very grateful that two independent British researchers not only did so, but used local data. Before the Ontario government condemns the province to higher electricity prices and more frequent blackouts on the basis of an obsolete air scare, they should take a careful look at what the evidence really says.
National Post, Thursday, February 5, 2004