Category:Evaluation

From Online-Partizipation
Jump to navigation Jump to search

Definition

Evaluation is the structured assessment of a participation process and its results against a predefined set of criteria. The aim of an evaluation is to assess to what degree a process is meeting certain ex-ante specified goals even though an evaluation might also uncover other (perhaps unintended) consequences. In the words of Rowe and Frewer (2005:94) "It is thus useful to differentiate evaluation from assessment, the former referring to the structured process of establishing the success or otherwise of an exercise against preset criteria, the latter referring to the relatively unstructured analysis of an exercise without preset effectiveness criteria, as occurs in the conducting of descriptive case studies."

The results of an evaluation do not only allow to judge the particular process in question but crucially helps to inform future participation projects as it contributes to the knowledge that ultimately enables to choose what formats and processes are best suited to achieve particular goals. Therefore evaluation efforts do not stop at simply judging certain criteria but are also interested in the factors that help or hinder meeting desired criteria.

Theories

Evaluation is a very broad concept that is not limited to a particular discipline. Theoretical frameworks for the evaluation of participation have come from various backgrounds including Political Science (in relation to political participation or democratic innovations), Sociology (e.g. related to collective action), Law (in relation to procedural justice or participatory rulemaking) or Environmental Studies. The basic question that any evaluation theory needs to address is to define and justify the analytical criteria that are of interest to be measured and on which the participation process is subsequently assessed upon.

There is a diversity of such approaches, dating back at least to Arnstein's ladder of participation from 1969 and again, reflecting various disciplinary perspectives. For more see Evaluation Frameworks. Evaluation approaches may be distinguished along various dimensions such as the ones below even though combinations are possible:

  1. initiator: are the commissioned by the initiator of a process or by an independent party
  2. goal: is the evaluation formative (in order to inform future processes) or summative (i.e. whether it reached its desired objective) (see Abelson & Gauvin (2006:3)
  3. nature of criteria: are the criteria based on more individual assessments, e.g. of particular stakeholders such as participants, or on more universal measures, e.g. such as number of people participating
  4. focus: process evaluations vs. result evaluations

As far as the evaluation of online participation is concerned, the field can still be considered under-theorised. The particular methods and tools used to measure the defined criteria are subject of the various strands of theories on research methodologies such as survey research or statistical inference.

Methods

Evaluation efforts make use of a great variety of methods, depending largely on the criteria that are subject to measurement. Among those found more often in evaluation reports are:

  • surveys, usually of participants
  • text and content analysis, e.g. of contributions of partipants
  • interviews, e.g. with administrators tasked with implementing such processes
  • literature reviews

selected research efforts

At the Heinrich-Heine-University Düsseldorf there are several research projects aimed at evaluating online participation, in particular at the local level. The project Effects of local online participation and their influencing factors in comparative perspective employs a comparative approach by conducting several nearly identical online participation processes and evaluating these systematically. In addition, several online participation processes were conducted within the university to cooperatively develop norms (see Cooperative norm setting online at the University (Düsseldorf)). There are also PhD projects in this area, e.g. Schoettle - The Gender Gap in Online-Participation.