Wednesday, 1 September

IPCC Management found lacking by the InterAcademy Council

An important report was released yesterday by the InterAcademy Council (IAC). This is multinational organization of science academies from around the world. The current eighteen-member board is composed of presidents of fifteen academies of science and equivalent organizations, including Professor Kurt Lambeck, past president of the Australian Academy of Science.

The United Nations and IPCC asked the InterAcademy Council (IAC) to review the processes and procedures of the IPCC and make recommendations for change. The report released yesterday is a result. The Review was carried out by a 12-member committee representing Academies of Sciences and Universities from USA, South Africa, Brazil, China, Netherlands, India, Mexico, UK, France and Malaysia. Another group of 12 people of similar stature reviewed the report and Australia's John Zillman, a former director of Australian Bureau of Meteorology and a former President of the World Meteorological Organization is amongst them. The Review process was monitored by Kurt Lambeck, a past president of the Australian Academy of Science, and Ralph Cicerone, President of the US National Academy of Sciences.

The Committee examined only the procedures and processes of the IPCC. It did not try to review climate change science or the validity of its representation in the IPCC assessments. The Report is 113 pages long and if you are interested you can download it from the IAC web site. While Report acknowledges many good things about the work of the IPCC, its most critical statement is that, since its inception in 1988, IPCC has failed to adapt to the changing conditions and the increased controversy on the climate change issue across the globe. This has to be seen as an indictment on the current management of the IPCC.

Sources of Data

I was under the impression that only peer-reviewed papers were considered for the IPCC in preparing their assessments. I learn that it started this way but it turned out that a significant amount of useful information was available in non-peer-reviewed sources. An analysis of the 14,000 references cited in the Third Assessment Report found that peer-reviewed journal articles comprised 84 percent of references in Working Group I, but only 59 percent of references in Working Group II and 36 percent of references in Working Group III. The Working Group 1 is the original IPCC brief assessing the physical aspects of the climate system and climate change. The Working Groups 2 and 3 deal with the vulnerability to climate change and policy options for mitigation, respectively. The IAC Report concludes that clearer guidelines and stronger mechanisms for enforcing them are needed in terms of including non-peer-reviewed literature.

Review Process

The IAC Committee notes that all IPCC Reports are subject to open review and they are reviewed extensively. For example, the Working Group II report of the Fourth Assessment received over 35000 expert comments and government comments. The sheer size of the commentary sometimes resulted in ignoring some of the reviews. The IAC Report recommends that (a) targeted reviews are sought and a range of views are included in these targets; and (b) a more rigorous review monitoring system is put in place so that all reviews are considered and addressed by the Lead Authors in the final product. The IAC Committee Report provides the IPCC Working Group II warning on the Himalaya Glaciers as an example of ignoring some of the reviewer comments, which would have increased the quality of the quality of the final product if they had been considered. I am not going to copy it here but I recommend you read it in the Report, it is on pp 23-24.

Influence of the Governments

The "Summary for the Policy makers" is probably the best-read part of an IPCC Assessment. These summaries are drafted by the scientists but negotiated with the government representatives into their final form. The negotiations occur in plenary sessions. These sessions last for several days and commonly end with an all-night meeting. Thus, the individuals with the most endurance or the countries that have large delegations can end up having the most influence on the report. The IAC Committee recommends that government inputs are provided as written comments prior to the plenary session.

Treatment of the Uncertainty

IPCC Scientists are asked to make predictions about the future. Different degrees of uncertainty are assigned to such predictions in the IPCC assessment reports. Since the authors of these reports extract these predictions from published research not from their own work, it is difficult for the to formally characterise uncertainty. This becomes especially difficult when you consider that 10000+ references are cited in each Assessment report. The standard IPCC procedure in assessing the uncertainty associated with a particular assertion in a particular reference appears to be assignment of subjective measures of confidence. The following table is included in the IAC Review Report to represent the IPCC procedure:

Terminology	Degree of Confidence in Being Correct
Very high confidence	At least 9 out of 10 chance of being correct
High confidence	About 8 out of 10 chance
Medium confidence	About 5 out of 10 chance
Low confidence	About 2 out of 10 chance
Very low confidence	Less than 1 out of 10 chance

I guess there is nothing wrong with stating subjective opinions, especially when a more rigorous uncertainty analysis is not possible. However, it may be misleading when these subjective opinions are presented as something which they are not. The IAC Review concludes that assigning probabilities to imprecise statements is not an appropriate way to characterize uncertainty. If the confidence scale is used in this way, conclusions will likely be stated so vaguely as to make them impossible to refute, and therefore statements of “very high confidence” will have little substantive value. It should be stated that the domain of the Working Group I is more amenable to statistical analysis and some predictions in their reports are the result of statistical analyses on measured trends.

Six of recommendations are made to improve this process. They are all good recommendations actually for anyone doing a study on future forecasting based on a review of others' work and need to be read and understood, IMHO, for all of us as we all are asked to make predictions about the future in one form or other. I quote the first one directly from the IAC Review Report:

"The IPCC uncertainty guidance provides a good starting point for characterizing uncertainty in the assessment reports. However, the guidance was not consistently followed in the fourth assessment, leading to unnecessary errors. For example, authors reported high confidence in statements for which there is little evidence, such as the widely-quoted statement that agricultural yields in Africa might decline by up to 50 percent by 2020. Moreover, the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented.

Scientific uncertainty is best communicated by indicating the nature, number, and quality of studies on a particular topic, as well as the level of agreement among studies. The level-of understanding scale is a convenient shorthand way of communicating this information in summary documents.

Recommendation: All Working Groups should use the qualitative level-of-understanding scale in their Summary for Policy Makers and Technical Summary, as suggested in IPCC’s uncertainty guidance for the Fourth Assessment Report. This scale may be supplemented by a quantitative probability scale, if appropriate" (IAC Report, p 48).

Management and Governance

The IAC Review is particularly critical about the way IPCC has been managed so far. Part of the problem, as they see it, is that the Panel has grown too quickly beyond its original terms of reference and the initial arrangements may not provide adequate support at the level of complexity and controversy facing the Panel. A permanent Executive Committee and a strengthened Secretariat is amongst the recommendations.

I was particularly interested in the comments on the Conflict of Interest. I quote directly from the Report since they express it in a much more elegant manner that I can do in trying to summarise it:

"A key governance feature of institutions that deal with broad public policy interests is the consideration of conflict of interest (NRC, 2002). The term “conflict of interest” refers to any financial or other interest that compromises the service of an individual by significantly impairing the individual’s objectivity or creating an unfair competitive advantage for any person or organization. Conflict of interest means something more than a strong view or bias—there must be an interest, ordinarily financial, that could be directly affected by the individual’s participation (NAS, 2003).

Many governmental and nongovernmental institutions that carry out scientific assessments or provide scientific advice have adopted conflict of interest and disclosure policies in order to assure the integrity of, and public confidence in, their results.....

The lack of a conflict of interest and disclosure policy for IPCC leaders and Lead Authors was a concern raised by a number of individuals who were interviewed by the Committee or provided written input. Questions about potential conflicts of interest, for example, have been raised about the IPCC Chair’s service as an advisor to, and board member of, for-profit energy companies (Booker and North, 2009; Pielke, 2010b), and about the practice of scientists responsible for writing IPCC assessments reviewing their own work. The Committee did not investigate the basis of these claims, which is beyond the mandate of this review. However, the Committee believes that the nature of the IPCC’s task (i.e., in presenting a series of expert judgments on issues of great societal relevance) demands that the IPCC pay special attention to issues of independence and bias to maintain the integrity of, and public confidence in, its results.

Recommendation: The IPCC should develop and adopt a rigorous conflict of interest policy that applies to all individuals directly involved in the preparation of IPCC reports, including senior IPCC leadership (IPCC Chair and Vice Chairs), authors with responsibilities for report content (i.e., Working Group Co-chairs, Coordinating Lead Authors, and Lead Authors), Review Editors, and technical staff directly involved in report preparation (e.g., staff of Technical Support Units and the IPCC Secretariat)." (IAC Report pp 45-46)

The rest of the Review Report is on implementing a more transparent communications policy for the IPCC and future issues such as the participation of developing nations and private sector in the IPCC deliberations; and the IPCC access to confidential data, e.g. commercial databases.

My Conclusions

I think this is a very strong report. When it was reported in the media, it was seen as an invitation for the IPCC Chair to resign

I welcome the introduction of more rigour and accountability in to the IPCC. The climate change is probably the biggest challenge facing the humanity. The IPCC is the leading agency in formulating responses to this challenge. Anything short of full competence and transparency on the part of the IPCC has the danger of producing doubt on the nature and the size of the climate change challenge. This IAC Review is a step in the right direction and it is a credit to the Review Panel. I guess we will wait and see what will follow.

Click here for the rest of the blog