Abelson and Gauvin (2006) - Assessing the Impacts of Public Participation
Abelson, Julia and Gauvin, François-Pierre (2006): Assessing the Impacts of Public Participation: Concepts, Evidence and Policy Implications. Ottawa: Public Invovlement Network (DOI: 10.1177/109019810002700610)
The Paper is an overview regarding the current knowledge about the state of evaluation of public participation and to what extent this knowledge (e.g. best practice) is deployed by evaluators on the decision makers side. To get valid results the authors emphasize the necessity to consider the impact of context-variables alongside the outcome/impact-related and process-related aspects of participation where success criteria (that are deployed in evaluations) are usually located.
The paper consists of three core elements (p. 2):
(1) A review of theoretical (and methodological) and conceptual literature concerning research agendas and frameworks. A “Conceptual Map of Public Participation Evaluation” (p. 18) proposes a structured perspective to locate a given evaluation effort.
(2) A review of English- and French-language empirical public participation literature collected over the 2000 – 2005 period
(3) The results of six qualitative interviews with policy makers that deal with the question in which ways and to which extent they evaluate public participation projects
Core Element (1)
- definition of public participation: "We have not adopted an explicit definition of public participation or involvement in this review. We wish to be clear at the outset of the paper that, by public, we are referring to ‘ordinary citizens’ rather than to organized groups of individuals or to individuals with special expertise in a policy area. We recognize that ordinary citizens are often part of many organized groups and may have a considerable degree of expertise in a variety of areas that they will bring to their participatory roles. But in our discussions about the public in this paper, it is neither their organizational affiliations nor their expertise that is the basis upon which their involvement is being sought." p2
- "Scholars within different fields of study are unanimous in their conclusions about the paucity of good quality research evidence about public participation and its effects." p4
Referring to Rosener (1981) challenges in evaluating public participation are seen in
- “the complexity and value-laden nature of public participation as a concept;
- the absence of widely held criteria for judging its success and failure;
- the lack of agreed-upon evaluation methods; and (4) the paucity of reliable measurement tools.” (p. 5)
A basic theoretical framework, developed by Webler (1995) that is widely used and/or modified by other authors concentrates on the deliberative aspect of public participation which can be understood as an interplay of fairness (equal distribution of opportunities) and competence (knowledge and understanding of the issue). (p. 5)
Referring to Beierle and Cayford (2002) the success of public participation can be assessed against five social goals: “(1) the incorporation of public values into decisions; (2) improvement of the substantive quality of decisions; (3) resolution of conflict among competing interests; (4) building trust in institutions; and (5) educating and informing the public.” (p. 6)
Against which criteria and by whom success is defined is left unclear in the current state or research. The tables 1 and 2 on pages eight and nine give a good impression of the multiple perspectives regarding this question. Especially the aspect of a valid operationalization and measurement of these criteria is highly unclear. This is true for the evaluation of the process as well as for the evaluation of the outcome of public participation: "The absence of properly tested measurement tools is another area in need of attention. [...] Moreover, few instruments had been validated or tested for reliability.” (p. 15)
Core Element (2)
Concerning the empirical public participation literature, a strain of outcome-orientated literature consistently reports viable effects of public participation such as: “Increased levels of interest in and knowledge of public issues; improved capacity for future public involvement; increased propensity for social bond formation and; improved trust of fellow citizens” (p. 20).
Delli Carpini, Cook and Jacobs (2004) conclude their extensive theory-based review that “the impact of deliberation […] is highly context dependent. It varies with the purpose of deliberation, the subject under discussion, who participates, the connection to authoritative decision makers, the rules governing interactions, the information provided, prior beliefs, substantive outcomes, and real-world conditions.” (p. 21).
Beierle and Cayford (2002) find – based on their meta-analysis of 239 cases of public involvement in environmental decision making – a strong association between the acceptance of the outcomes and attributes of the process (agencies responsiveness, high quality of deliberation, a minimum amount of control over the process by the participants).
Core Element (3)
Only a (small) minority of decision makers (the data is based on a survey conducted in the health system) deploys scientific knowledge to formally evaluate public participation projects (about 10%), whereas about half of the interviewed decision makers only deploy informal forms of evaluation and 20% leave the public participation projects unevaluated.
The key-informant interviews show a low commitment to any form of science-based and formal evaluation of public participation projects by mostly using participants’ satisfaction questionnaires if any evaluation is conducted at all. So the barriers to a more systematic deployment of public participation evaluation seem be rooted in a lack of commitment especially on the senior management level in governments itself, what leads the authors to the conclusion: "If you’re not serious about public participation, why get serious about evaluating it?" (p. 36).