Institute for Public Information Management

fortiss GmbH

An-Institut Technische Universität München

Measuring the promise of Open Data:

Development of the Impact Monitoring Framework

CeDEM16, Krems, 2016-05-18

Marcus M. Dapp, dapp@fortiss.org, @digisus

Institute for Public Information Management

Berlin, 2016-04-18CeDEM16, KRems2

Prof. Dr. Helmut Krcmar

Lehrstuhl für Wirtschaftsinformatik

Technische Universität München

Scientific Director fortiss

Dr. Marcus M. Dapp

fortiss – An-Institut der TU München

Guerickestraße 25

80805 München

dapp@fortiss.org | @digisus | 089 3603522 19

• eGovernment MONITOR (DACH)

• Open Data Potential für Deutschland

www.dataportals.org

• Mostly qualitative• Anecdotal evidence

• Case studies

• Some ranking schemes• Open Data Barometer – expert opinion

• Open Data Census – expert and non-expert („crowd“) opinion

• Open Government Partnership – national action plans

• Gap identified• Ubaldi (OECD): 96% of governments (N=16) have NOT “adopted a

methodology to measure return on investment on open government.”

Measuring Open Data Impact (I)

• … is hard• Easy-to-measure metrics (data downloads) do not tell anything about impact

and impactful activities (e.g. startups) are not easy to measure.

• Hard to trace links between cause and effect

• Long time lag between cause and effect

• Complex relationships between causes and effects

• … is needed• To support the effective and efficient use of resources

• To focus on high-impact Open Data activities

• To increase the success of new Open Data initiatives

Measuring Open Data Impact (II)

• While ROI approach measures only profit, SROI includes the economic, socio-economic, and social value of an enterprise. (introduced to measure social value of philanthropic investments)

• “Theory of change”: resources (input), transformed into controllable results (output), can foster indirect activities (outcome) and lead to value-creating consequences (impact). Impact map and indicators used to capture these causal relationships across all stakeholders.

• IDEA: Impact Monitoring Framework combines the SROI technique with open data impact literature to suggest a new approach.

Concept: Social Return on Investment

A „Theory of Change“ for Open (Gov) Data

INPUT all resources such as money, people, equipment, and facilities used by an organization to publish

open data, plus all pre-existing native data. “native data” because it is a natural part of the

organization and available e.g. in proprietary formats and software tools not yet ready to be published

as “raw open data”.

OUTPUT direct and tangible deliverables produced by the organization; mainly the setup and operation of an

open data portal. Quality of portals differs widely today: quality of metadata, completeness of

datasets, platform’s accessibility and visibility, usability and comprehensibility, timeliness, value and

usefulness as well as granularity.

OUTCOME all direct and indirect consequences of certain output actions by the organization; all activities by

the re-users of open data: engineers, entrepreneurs, citizens, journalists, scientists, artists or

administrators re-using the available open data in some form. Activities include hackathons,

visualizations, web and mobile applications, new open-data-based business models, data journalism,

research projects etc. Uptake of open data cannot be controlled by the organization releasing the

data, it can only support and encourage data re-use.

IMPACT outcome adjusted for the effects that would have occurred without publishing data by the

organization. Only results caused by releasing open data are counted. In practice, it is not easy to

measure the impact of specific output. Guidelines suggest using comparison groups or benchmarks.

value

chain

Data categories(G8 Charta)

Impact Monitoring Framework (examples)

Discussion

• Model critique• SROI is very costly (teim, data) to be meaningful

• We need to test the framework• We are looking for „open data projects/initiatives“ as cases for empirical

testing

• Do re-use activity patterns and data category correlate?• If yes: Coming up with generic activity types most suited for a data category

would help governments and re-users to be more effective.

• Complete the framework by adding monetization• SROI ratio = Social impact (monetized) / Investment (monetized), e.g. 1:3