Lippa et al. (2008) - DEMO-net: D 13.3 eParticipation Evaluation and Impact

From Online-Partizipation
Jump to navigation Jump to search

Lippa, B.; Aichholzer, G.; Allhutter, D.; Freschi, A.; Macintosh, A.; Moss, G.; Westholm, H. (2008): DEMO-net: D 13.3 eParticipation Evaluation and Impact. Bremen/ Leeds: DEMO-net Consortium


The booklet proposes how to evaluate “eParticipation”. At its core it develops a theoretical framework of evaluation criteria that is structured by a “layered model of eParticipation evaluation” which differentiates the evaluation process in three perspectives: the project perspective, the socio-technical perspective and the democratic perspective. Every perspective inherits specific analytical criteria, that each are constituted by sub-criteria. Operational definitions of the concepts, suggestions concerning the measurement, and a methodological review are given as well.


Aside from the theoretical framework the authors contribute to the fact that “context matters” and highly suggest to systematically consider the following factors of the participation context before deploying any kind of evaluation:

  • governmental levels
  • issues/types of decisions
  • issue characteristics
  • tools and methods used
  • scope of the exercise
  • level of engagement
  • stage in decision-making
  • actors
  • degree of institutionalisation /formalisation of the procedure
  • project’s history
  • political culture

The theoretical framework itself is structured by three main perspectives of evaluation:

  1. The Project perspective, that deals with stakeholders’ aims, methods and expectations of public engagement
  2. The Social-technical perspective, that deals with the deployed technical tools of eParticipation and focuses on the public take-up, usefulness and acceptability
  3. The Democratic perspective, that aims at the outcomes and impacts an eParticipation project or eParticipation as a whole have to offer to participants, democratic values and/or the democratic system as a whole

From each of these three perspectives the authors derive specific analytical criteria against which the success of an eParticipation project could be measured. For each of the criteria/sub-criteria indicators and measurement-methods are suggested, that range from surveys, media reviews to content analysis and many more. The methods themselves are elaborated and discussed from a methodological standpoint later on in the text. The theoretical framework (criteria/sub-criteria/indicators) presents itself as followed:

  • The Project perspective
    • Engaging with a wider audience
      • Outreach and promotion
        • Media coverage
      • Inclusiveness
        • Number and social profile of involved people
        • Identification of participation barriers
        • Incorporation of multiple perspectives in service design, technology production and use
      • Community development
        • Individual participation and networking
        • Connection patterns among different networks of interest
    • Obtaining better-informed opinions
      • Information
        • Relevance and quality of the provided information
      • Learning
        • Scope of expertise (before and after the participation process)
    • Scope of deliberation
        • Extent of interaction among participants
        • Extent of rationality involved in the participation process
    • Effectiveness
      • Cost effectiveness
        • Cost savings in providing, aggregating and evaluating input
      • Time effectiveness
        • Time savings in providing, aggregating and evaluating input
    • Feedback
        • Feedback content and quality
        • Response rates and timeliness of response
        • Participants’ satisfaction with feedback
        • Public availability of contributions of other participants; proportion of submissions posted for public viewing
    • Process quality
        • User and stakeholder identified areas for enhancement
        • Gap analysis against good practice features of processes in practice elsewhere
        • Harmonization of work-practices of authority and eParticipation processes
    • Sustainability
        • Level of key stakeholder support
        • Stakeholder perception of continuity barriers
        • Level of institutionalization of education and training for government officials
  • Socio-technical perspective
    • Social acceptability
      • Trust and security
        • the information is presented accurate, complete and reliable
        • the information users have provided is handled in a secure manner
        • users are confident in the steps taken
        • data handling procedures are in compliance with relevant legislation or guidelines
      • Relevance and legitimacy
        • tool meets a purpose relevant to individual users’ and community’s needs
        • the content and surrounding processes are relevant to that purpose
    • Usefulness
      • Accessibility
        • level of compliance with Web Accessibility Initiative (WAI) content guidelines is sufficient to meet the needs of users with disabilities
        • alternative access over public access points, mobile devices is possible
        • identification of access barriers
      • Appeal and usage
        • the take-up is inline with expectations
        • intended users are satisfied with the tool and want to use it; willingness to return to the site
        • identification of usage barriers
        • number of users, extent of their use of the tool
      • Content clarity
        • users understand what the content means in the context of the task or situation
      • Responsiveness
        • the tool allows to answer the user’s questions quickly and effectively
        • site provides contact information, FAQs, search functions
      • Interaction
        • the tool supports the level of interaction required by the process
      • Good practice
        • level of consistency with current developments, good practice guidelines and standards in the field (e.g., interoperability standards)
    • Usability
      • Navigation and organisation
        • sufficient and consistent information about users’ current position on the site, path taken, and available options
      • Efficiency and flexibility
        • time needed to perform a task
        • appropriate short-cuts for doing repetitive or familiar tasks
      • Error recovery
        • users can undo previous actions
        • users are guided effectively on the correct procedure and can continue the task without distraction or hesitation
  • The democratic perspective
    • Representation
      • Fit with legal framework
        • Legal stipulation for participation procedure
      • Integration with “offline” participation processes
        • Matrix on where eParticipation project or tool can interlink with “offline” policy processes
    • Engagement
      • Information
        • Availability of information on democratic processes and rules (of eParticipation)
      • Knowledge
        • Knowledge about existing initiatives
        • Increase in knowledge about democratic processes and rules (of e-)Participation
      • Participation
        • Extent of participation: access numbers, time, number of initiatives
        • Active membership in networks or interest groups
        • Level of involvement of citizens in identifying thematic issues to be dealt with via eParticipation tools (top-down, bottom-up)
        • Social capital building
    • Transparency
        • Publication of interim and/or final results of participation process
        • Transparency on how decisions were negotiated
        • Public discussion on final results with involved actors
    • Conflict and consensus
        • Identification of “pros and cons” (to avoid a “consensus manufactory”); level of agreement amongst participants
        • Participation policies (e.g., moderation, etc.)
    • Political equality
      • Pluralism, openness of participation process
        • Number of relevant target group(s) in relation to actual number of participants
        • Identification of barriers to participation or barriers to active citizenship
    • Community control
      • Participant satisfaction
        • Satisfaction index
      • Impact on decision-making process
        • Level of departmental and jurisdictional integration16, accountability
        • Level of documentation of generated results
        • Identification of differing interests and policy frames
        • Policy outcomes in relation to different interests articulated by stakeholders
        • Impact in different stages of decision-making; identification of relevant “points of cooperation” (e.g., according to policy-life cycle)


The authors conclude that given the restrains on resources and external/contextual factors only a fraction of the framework might be applied in reality, but it should none the less serve as a guideline to make “make more balanced and intelligent decisions in evaluating eParticipation” (p. 49).